Metrics scheduler closed

Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110320892
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-1
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321002
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-2
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321112
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-3
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321221
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-4
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321330
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-5
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321438
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-6
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321548
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-7
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110321656
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-8
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Reading segment sizes from 6 servers for table: amxordersntt_REALTIME with timeout:
30000ms
Finished reading information for table: amxordersntt_REALTIME
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: df1e6583_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: df1e6583_TASK. Event session ID: 3000a812cfc0036
Event df1e6583_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event df1e6583_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event df1e6583_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event df1e6583_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event df1e6583_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event df1e6583_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event df1e6583_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event df1e6583_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::df1e6583_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::df1e6583_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/3d698599-21cf-4481-844c-
bd08e2a3c9a8=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/68ecfab8-f842-43e8-9d1e-
79bec3636751=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/8da8c0a2-9cb6-4c92-a1e5-
31ed39c3e818=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/3d698599-21cf-4481-844c-
bd08e2a3c9a8=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/68ecfab8-f842-43e8-9d1e-
79bec3636751=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/8da8c0a2-9cb6-4c92-a1e5-
31ed39c3e818=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322081, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event df1e6583_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::df1e6583_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322105, took 0 ms
Event PinotCluster::TASK::df1e6583_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::df1e6583_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322081, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event df1e6583_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322052 took 60 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 60 ms
for event df1e6583_DEFAULT
Event PinotCluster::TASK::df1e6583_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::df1e6583_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322109, took 20 ms
Event df1e6583_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322052 took 78 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 78 ms
for event df1e6583_TASK
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event df1e6583_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event df1e6583_DEFAULT
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event df1e6583_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event df1e6583_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event df1e6583_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event df1e6583_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event df1e6583_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event df1e6583_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event df1e6583_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event df1e6583_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event df1e6583_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event df1e6583_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event df1e6583_TASK
END: Invoking TASK controller pipeline for event MessageChange::df1e6583_TASK for
cluster PinotCluster, took 149 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: e0b97209_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event df1e6583_DEFAULT
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
 client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
 ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322229
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios--2147483648
Event PinotCluster::TASK::e0b97209_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::e0b97209_TASK : Refreshed 11 property IdealState took 28
ms. Selective: true
Event e0b97209_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e0b97209_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e0b97209_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event PinotCluster::TASK::e0b97209_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
 ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
overwrite existing constraint-value. old-value: null, new-value: 100000
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322233
Event PinotCluster::TASK::e0b97209_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-
info,id=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
at
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithR
epository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam
icMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec
t(DefaultMBeanServerInterceptor.java:895) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean
(DefaultMBeanServerInterceptor.java:320) ~[?:?]
at
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer
.java:523) ~[?:?]
at
org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo
(AppInfoParser.java:64) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:814) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:665) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:646) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:626) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createC
onsumer(KafkaPartitionLevelConnectionHandler.java:83) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(
KafkaPartitionLevelConnectionHandler.java:70) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStre
amMetadataProvider.java:59) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadata
Provider(KafkaConsumerFactory.java:38) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(St
reamMetadataProvider.java:91) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:70) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:31) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGrou
pMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.realtime.MissingConsumingSegmentFinder.<init
>(MissingConsumingSegmentFinder.java:79) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.updateSegmentMetrics(Segment
StatusChecker.java:375) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:124) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:66) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processT
ables(ControllerPeriodicTask.java:116) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(
ControllerPeriodicTask.java:79) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTas
kScheduler.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
ScheduledThreadPoolExecutor.java:305) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1136) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-0
START: InstanceMessagesCache.refresh()
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event df1e6583_DEFAULT
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
 sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322241
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-1
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
 sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322246
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-2
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event df1e6583_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event df1e6583_DEFAULT
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
 sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322255
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-3
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event df1e6583_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event df1e6583_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event df1e6583_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event df1e6583_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event df1e6583_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::df1e6583_DEFAULT
for cluster PinotCluster, took 209 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 209 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: e0b97209_DEFAULT. Event session ID: 3000a812cfc0036
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
 default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
 ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322265
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-4
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
 default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
 ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322270
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-5
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
 default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
 ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322277
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-6
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event PinotCluster::TASK::e0b97209_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e0b97209_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322255, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
 client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
 ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322284
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-7
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : Refreshed 11 property IdealState
took 24 ms. Selective: true
Event e0b97209_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e0b97209_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e0b97209_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
 security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322289
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-8
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
START: InstanceMessagesCache.refresh()
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
 sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322295
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-9
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
 sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322302
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-10
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event PinotCluster::TASK::e0b97209_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e0b97209_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322281, took 24 ms
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
Event e0b97209_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322201 took 104 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 104 ms
for event e0b97209_TASK
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
 sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322307
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-11
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
 sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322313
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-12
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
 sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322320
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-13
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event e0b97209_TASK
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e0b97209_TASK
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
 reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322325
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-14
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : 0 properties refreshed from ZK.
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322311, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e0b97209_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e0b97209_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322336, took 0 ms
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
 max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322337
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-15
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event e0b97209_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322261 took 80 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 80 ms
for event e0b97209_DEFAULT
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
 internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322343
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-16
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
 internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322349
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-17
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
 internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322355
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-18
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event e0b97209_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e0b97209_DEFAULT
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
 group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
 ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322363
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-19
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
 group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
 ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322369
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-20
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
 group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
 ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322376
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-21
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event e0b97209_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event e0b97209_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e0b97209_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e0b97209_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e0b97209_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e0b97209_TASK
END: Invoking TASK controller pipeline for event MessageChange::e0b97209_TASK for
cluster PinotCluster, took 177 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 5ccfbf39_TASK. Event session ID: 3000a812cfc0036
Event 5ccfbf39_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 5ccfbf39_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 5ccfbf39_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 5ccfbf39_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event PinotCluster::TASK::5ccfbf39_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::5ccfbf39_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
START: InstanceMessagesCache.refresh()
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
 sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322385
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-22
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
 sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322390
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-23
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Reading segment sizes from 7 servers for table: clickstream_REALTIME with timeout:
30000ms
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event e0b97209_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e0b97209_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e0b97209_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Finished reading information for table: clickstream_REALTIME
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
 metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322424
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login--2147483648
Event e0b97209_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Event PinotCluster::TASK::5ccfbf39_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::5ccfbf39_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322403, took 24 ms
Skipping auto SSL server validation since it's not configured.
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
 metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322428
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-
info,id=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
at
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithR
epository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam
icMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec
t(DefaultMBeanServerInterceptor.java:895) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean
(DefaultMBeanServerInterceptor.java:320) ~[?:?]
at
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer
.java:523) ~[?:?]
at
org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo
(AppInfoParser.java:64) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:814) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:665) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:646) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:626) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createC
onsumer(KafkaPartitionLevelConnectionHandler.java:83) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
 at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(
KafkaPartitionLevelConnectionHandler.java:70) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStre
amMetadataProvider.java:59) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadata
Provider(KafkaConsumerFactory.java:38) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(St
reamMetadataProvider.java:91) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:70) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:31) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGrou
pMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.realtime.MissingConsumingSegmentFinder.<init
>(MissingConsumingSegmentFinder.java:79) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.updateSegmentMetrics(Segment
StatusChecker.java:375) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:124) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:66) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processT
ables(ControllerPeriodicTask.java:116) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(
ControllerPeriodicTask.java:79) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTas
kScheduler.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
ScheduledThreadPoolExecutor.java:305) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1136) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-0
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating the new best possible assignment.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
 ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322436
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-1
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
 ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

Computing BestPossibleMapping for positionamxgpx_REALTIME

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a
known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322446
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-2
Event PinotCluster::TASK::5ccfbf39_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::5ccfbf39_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322427, took 20 ms
Event 5ccfbf39_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322378 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 5ccfbf39_TASK
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
 sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322454
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-3
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
 sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

Computing BestPossibleMapping for clickstreamd_REALTIME

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a
known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322461
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-4
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 5ccfbf39_TASK
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
 sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms

for event 5ccfbf39_TASK
The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a
known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110322468
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-5
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
Computing BestPossibleMapping for amxordersgpx_REALTIME
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Reading segment sizes from 1 servers for table: clientlogin_REALTIME with timeout:
30000ms
Finished reading information for table: clientlogin_REALTIME
Finish processing 3/3 tables in task: SegmentStatusChecker
[TaskRequestId: auto] Finish running task: SegmentStatusChecker in 3758ms
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
84 ms for event e0b97209_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 5ccfbf39_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 5ccfbf39_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 5ccfbf39_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 5ccfbf39_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 5ccfbf39_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 5ccfbf39_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::5ccfbf39_TASK
for cluster PinotCluster, took 141 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event e0b97209_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event e0b97209_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e0b97209_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event e0b97209_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event e0b97209_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event e0b97209_DEFAULT : Sending Message 70daf4c5-8d95-4f97-980e-dd218e611db8 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__31__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event e0b97209_DEFAULT : Sending Message 16708d2d-cbe6-4dd3-bb5c-dcffa2355e7b to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__31__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event e0b97209_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e0b97209_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e0b97209_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::e0b97209_DEFAULT
for cluster PinotCluster, took 292 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 182 ms
TotalProcessed time for event: MessageChange took: 292 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 5ccfbf39_DEFAULT. Event session ID: 3000a812cfc0036
Event 5ccfbf39_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 5ccfbf39_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 5ccfbf39_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 5ccfbf39_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 0816e115_TASK. Event session ID: 3000a812cfc0036
Event 0816e115_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 0816e115_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 0816e115_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 0816e115_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::0816e115_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : Refreshed 1 property
ClusterConstraint took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::0816e115_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322580, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 1 ms.
Event 5ccfbf39_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::5ccfbf39_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322611, took 0 ms
Event PinotCluster::TASK::0816e115_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::0816e115_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322583, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 5ccfbf39_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322553 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event 5ccfbf39_DEFAULT
Event PinotCluster::TASK::0816e115_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::0816e115_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322613, took 20 ms
Event 0816e115_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322555 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 0816e115_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 5ccfbf39_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 5ccfbf39_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 17
ms for event 0816e115_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0816e115_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 5ccfbf39_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 5ccfbf39_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 5ccfbf39_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 0816e115_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 0816e115_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 0816e115_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 0816e115_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0816e115_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0816e115_TASK
END: Invoking TASK controller pipeline for event MessageChange::0816e115_TASK for
cluster PinotCluster, took 149 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 7ba8b277_TASK. Event session ID: 3000a812cfc0036
Event 7ba8b277_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 7ba8b277_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 7ba8b277_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 7ba8b277_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::7ba8b277_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::7ba8b277_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 5ccfbf39_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 5ccfbf39_DEFAULT
Event PinotCluster::TASK::7ba8b277_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7ba8b277_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322729, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 5ccfbf39_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 5ccfbf39_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 5ccfbf39_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 5ccfbf39_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 5ccfbf39_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 5ccfbf39_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 5ccfbf39_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::5ccfbf39_DEFAULT for cluster PinotCluster, took 215 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 422 ms
TotalProcessed time for event: IdealStateChange took: 215 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: e82714ef_DEFAULT. Event session ID: 3000a812cfc0036
Event e82714ef_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event e82714ef_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e82714ef_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e82714ef_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::7ba8b277_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7ba8b277_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322754, took 20 ms
Event 7ba8b277_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322704 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 7ba8b277_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 7ba8b277_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ba8b277_TASK
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322794, took 37 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e82714ef_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e82714ef_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110322831, took 0 ms
Event e82714ef_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322768 took 69 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 69 ms
for event e82714ef_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event e82714ef_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e82714ef_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
65 ms for event 7ba8b277_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7ba8b277_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7ba8b277_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 7ba8b277_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ba8b277_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ba8b277_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::7ba8b277_TASK
for cluster PinotCluster, took 156 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: e82714ef_TASK. Event session ID: 3000a812cfc0036
Event e82714ef_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e82714ef_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e82714ef_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e82714ef_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 1 ms
Event PinotCluster::TASK::e82714ef_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e82714ef_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event e82714ef_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e82714ef_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e82714ef_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::e82714ef_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e82714ef_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322886, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event e82714ef_DEFAULT
Event PinotCluster::TASK::e82714ef_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e82714ef_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110322916, took 20 ms
Event e82714ef_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322860 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event e82714ef_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 17
ms for event e82714ef_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event e82714ef_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event e82714ef_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event e82714ef_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e82714ef_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event e82714ef_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e82714ef_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event e82714ef_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e82714ef_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e82714ef_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END: Invoking DEFAULT controller pipeline for event MessageChange::e82714ef_DEFAULT
for cluster PinotCluster, took 217 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 168 ms
TotalProcessed time for event: MessageChange took: 217 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Controller stopping periodical rebalance timer at period 9223372036854775807
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 7ba8b277_DEFAULT. Event session ID: 3000a812cfc0036
Event 7ba8b277_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 7ba8b277_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 7ba8b277_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 7ba8b277_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event e82714ef_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e82714ef_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e82714ef_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e82714ef_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e82714ef_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e82714ef_TASK
END: Invoking TASK controller pipeline for event MessageChange::e82714ef_TASK for
cluster PinotCluster, took 147 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323011, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7ba8b277_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7ba8b277_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323037, took 0 ms
Event 7ba8b277_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110322985 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event 7ba8b277_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 7ba8b277_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 7ba8b277_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 7ba8b277_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 7ba8b277_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7ba8b277_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ba8b277_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event 7ba8b277_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 7ba8b277_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 56 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 7ba8b277_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7ba8b277_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 7ba8b277_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ba8b277_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 1 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 7ba8b277_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ba8b277_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7ba8b277_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::7ba8b277_DEFAULT for cluster PinotCluster, took 205 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 390 ms
TotalProcessed time for event: CurrentStateChange took: 205 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 597c16d6_DEFAULT. Event session ID: 3000a812cfc0036
Event 597c16d6_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 597c16d6_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 597c16d6_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 597c16d6_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 597c16d6_TASK. Event session ID: 3000a812cfc0036
Event 597c16d6_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 597c16d6_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 597c16d6_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 597c16d6_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::597c16d6_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::597c16d6_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/577be9c1-4ba0-42d6-939c-
11c8e75c3524=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/831859f4-9c75-4e6e-b5f7-
820965272ba6=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/375f4c25-4300-44c9-9159-
349356233eee=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/577be9c1-4ba0-42d6-939c-
11c8e75c3524=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/831859f4-9c75-4e6e-b5f7-
820965272ba6=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/375f4c25-4300-44c9-9159-
349356233eee=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323371, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 597c16d6_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::597c16d6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323396, took 0 ms
Event PinotCluster::TASK::597c16d6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::597c16d6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323371, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 597c16d6_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323344 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 59 ms
for event 597c16d6_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::597c16d6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::597c16d6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323400, took 20 ms
Event 597c16d6_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323344 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event 597c16d6_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 597c16d6_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 597c16d6_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 597c16d6_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 597c16d6_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 70 ms for event 597c16d6_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 597c16d6_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 597c16d6_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
68 ms for event 597c16d6_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 597c16d6_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 597c16d6_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 597c16d6_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 597c16d6_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 597c16d6_TASK
END: Invoking TASK controller pipeline for event MessageChange::597c16d6_TASK for
cluster PinotCluster, took 164 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 390464c7_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event 597c16d6_DEFAULT
Event PinotCluster::TASK::390464c7_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::390464c7_TASK : Refreshed 11 property IdealState took 24
ms. Selective: true
Event 390464c7_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 390464c7_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 390464c7_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::390464c7_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::390464c7_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 597c16d6_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 597c16d6_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 597c16d6_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 597c16d6_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 597c16d6_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 597c16d6_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 597c16d6_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 597c16d6_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::597c16d6_DEFAULT
for cluster PinotCluster, took 223 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 223 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 390464c7_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::390464c7_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::390464c7_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event 390464c7_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 390464c7_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 390464c7_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::390464c7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::390464c7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323557, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::390464c7_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::390464c7_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::390464c7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::390464c7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323583, took 25 ms
Event 390464c7_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323508 took 102 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 102 ms
for event 390464c7_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 24
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 390464c7_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 390464c7_TASK
Event PinotCluster::DEFAULT::390464c7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::390464c7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323613, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 390464c7_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::390464c7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::390464c7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323638, took 0 ms
Event 390464c7_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323567 took 76 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 76 ms
for event 390464c7_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 390464c7_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 390464c7_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 390464c7_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 390464c7_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 390464c7_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 390464c7_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 390464c7_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 390464c7_TASK
END: Invoking TASK controller pipeline for event MessageChange::390464c7_TASK for
cluster PinotCluster, took 175 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 6c7c4ef7_TASK. Event session ID: 3000a812cfc0036
Event 6c7c4ef7_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 6c7c4ef7_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 6c7c4ef7_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 6c7c4ef7_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::6c7c4ef7_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::6c7c4ef7_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 390464c7_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 390464c7_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 390464c7_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 390464c7_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::6c7c4ef7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::6c7c4ef7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323709, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 24 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::6c7c4ef7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::6c7c4ef7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323739, took 20 ms
Event 6c7c4ef7_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323683 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event 6c7c4ef7_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 32
ms for event 6c7c4ef7_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 6c7c4ef7_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
95 ms for event 390464c7_DEFAULT
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 6c7c4ef7_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 6c7c4ef7_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 6c7c4ef7_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 6c7c4ef7_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 6c7c4ef7_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 6c7c4ef7_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::6c7c4ef7_TASK
for cluster PinotCluster, took 164 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 390464c7_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 390464c7_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 390464c7_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 390464c7_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 390464c7_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 390464c7_DEFAULT : Sending Message 61aad771-60a0-4ce9-8a46-850acfde08bb to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__21__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 390464c7_DEFAULT : Sending Message ce842c18-b321-4ec9-8af8-0d82422ec0c1 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__21__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 390464c7_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 390464c7_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 390464c7_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::390464c7_DEFAULT
for cluster PinotCluster, took 308 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 198 ms
TotalProcessed time for event: MessageChange took: 308 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 6c7c4ef7_DEFAULT. Event session ID: 3000a812cfc0036
START AsyncProcess: DEFAULT::PersistAssignmentStage
Event 6c7c4ef7_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Event 6c7c4ef7_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 6c7c4ef7_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 6c7c4ef7_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 344e0260_TASK. Event session ID: 3000a812cfc0036
Event 344e0260_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 344e0260_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 344e0260_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 344e0260_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::344e0260_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::344e0260_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323902, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 6c7c4ef7_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::6c7c4ef7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110323932, took 0 ms
Event 6c7c4ef7_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323875 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event 6c7c4ef7_DEFAULT
Event PinotCluster::TASK::344e0260_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::344e0260_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323906, took 34 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 6c7c4ef7_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6c7c4ef7_DEFAULT
Event PinotCluster::TASK::344e0260_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::344e0260_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110323940, took 20 ms
Event 344e0260_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110323879 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event 344e0260_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 344e0260_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 344e0260_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 6c7c4ef7_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 6c7c4ef7_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6c7c4ef7_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 344e0260_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 344e0260_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 344e0260_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 344e0260_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 344e0260_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 344e0260_TASK
END: Invoking TASK controller pipeline for event MessageChange::344e0260_TASK for
cluster PinotCluster, took 158 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: d3d02bdb_TASK. Event session ID: 3000a812cfc0036
Event d3d02bdb_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event d3d02bdb_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event d3d02bdb_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event d3d02bdb_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::d3d02bdb_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 6c7c4ef7_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::d3d02bdb_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 6c7c4ef7_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 6c7c4ef7_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6c7c4ef7_DEFAULT
Event PinotCluster::TASK::d3d02bdb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d3d02bdb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110324062, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 6c7c4ef7_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6c7c4ef7_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 6c7c4ef7_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6c7c4ef7_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6c7c4ef7_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::6c7c4ef7_DEFAULT for cluster PinotCluster, took 221 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 459 ms
TotalProcessed time for event: IdealStateChange took: 221 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 52447823_DEFAULT. Event session ID: 3000a812cfc0036
Event 52447823_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 52447823_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 52447823_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 52447823_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::52447823_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::52447823_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::d3d02bdb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d3d02bdb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110324087, took 21 ms
Event d3d02bdb_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110324037 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event d3d02bdb_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 22
ms for event d3d02bdb_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d3d02bdb_TASK
Event PinotCluster::DEFAULT::52447823_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52447823_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110324123, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 52447823_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::52447823_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52447823_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110324147, took 0 ms
Event 52447823_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110324096 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 52447823_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 52447823_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 52447823_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event d3d02bdb_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d3d02bdb_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event d3d02bdb_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event d3d02bdb_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d3d02bdb_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d3d02bdb_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::d3d02bdb_TASK
for cluster PinotCluster, took 150 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 52447823_TASK. Event session ID: 3000a812cfc0036
Event 52447823_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 52447823_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 52447823_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 52447823_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::52447823_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::52447823_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 52447823_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 52447823_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52447823_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::52447823_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::52447823_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110324212, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 24 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 52447823_DEFAULT
Event PinotCluster::TASK::52447823_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::52447823_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110324236, took 21 ms
Event 52447823_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110324187 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 52447823_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 52447823_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52447823_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 52447823_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 52447823_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52447823_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 52447823_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52447823_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 52447823_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52447823_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52447823_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::52447823_DEFAULT
for cluster PinotCluster, took 211 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 172 ms
TotalProcessed time for event: MessageChange took: 211 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: d3d02bdb_DEFAULT. Event session ID: 3000a812cfc0036
Event d3d02bdb_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event d3d02bdb_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event d3d02bdb_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event d3d02bdb_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 52447823_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 52447823_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 52447823_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 52447823_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52447823_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52447823_TASK
END: Invoking TASK controller pipeline for event MessageChange::52447823_TASK for
cluster PinotCluster, took 142 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110324333, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event d3d02bdb_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d3d02bdb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110324357, took 0 ms
Event d3d02bdb_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110324308 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event d3d02bdb_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event d3d02bdb_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d3d02bdb_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event d3d02bdb_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event d3d02bdb_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d3d02bdb_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d3d02bdb_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event d3d02bdb_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event d3d02bdb_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 62 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event d3d02bdb_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event d3d02bdb_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event d3d02bdb_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d3d02bdb_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 1 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event d3d02bdb_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d3d02bdb_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d3d02bdb_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::d3d02bdb_DEFAULT for cluster PinotCluster, took 210 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 389 ms
TotalProcessed time for event: CurrentStateChange took: 210 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bb45ee4b_DEFAULT. Event session ID: 3000a812cfc0036
Event bb45ee4b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event bb45ee4b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event bb45ee4b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event bb45ee4b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bb45ee4b_TASK. Event session ID: 3000a812cfc0036
Event bb45ee4b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event bb45ee4b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event bb45ee4b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event bb45ee4b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::bb45ee4b_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::bb45ee4b_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/24162c2d-2abc-49cc-ac73-
fcbe5833f422=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/1b63a470-edaf-4d7c-a168-
7391dcf7eeac=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/7237ca54-2930-49aa-a289-
388259406f02=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 26
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/24162c2d-2abc-49cc-ac73-
fcbe5833f422=-101}
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 26
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110325538, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event bb45ee4b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bb45ee4b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110325564, took 0 ms
Event PinotCluster::TASK::bb45ee4b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bb45ee4b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325539, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event bb45ee4b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110325506 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event bb45ee4b_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::bb45ee4b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bb45ee4b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325569, took 20 ms
Event bb45ee4b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110325506 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event bb45ee4b_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event bb45ee4b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bb45ee4b_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event bb45ee4b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bb45ee4b_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event bb45ee4b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event bb45ee4b_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bb45ee4b_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
27 ms for event bb45ee4b_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
71 ms for event bb45ee4b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bb45ee4b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bb45ee4b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event bb45ee4b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bb45ee4b_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bb45ee4b_TASK
END: Invoking TASK controller pipeline for event MessageChange::bb45ee4b_TASK for
cluster PinotCluster, took 176 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 7ec6beb1_TASK. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::7ec6beb1_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::7ec6beb1_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 7ec6beb1_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 7ec6beb1_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 7ec6beb1_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::7ec6beb1_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::7ec6beb1_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event bb45ee4b_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event bb45ee4b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bb45ee4b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event bb45ee4b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bb45ee4b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event bb45ee4b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bb45ee4b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bb45ee4b_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::bb45ee4b_DEFAULT
for cluster PinotCluster, took 232 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 232 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 7ec6beb1_DEFAULT. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Event PinotCluster::TASK::7ec6beb1_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7ec6beb1_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325723, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : Refreshed 11 property IdealState
took 18 ms. Selective: true
Event 7ec6beb1_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 7ec6beb1_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 7ec6beb1_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::7ec6beb1_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7ec6beb1_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325747, took 21 ms
Event 7ec6beb1_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110325682 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event 7ec6beb1_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 7ec6beb1_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ec6beb1_TASK
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110325781, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7ec6beb1_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7ec6beb1_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110325805, took 0 ms
Event 7ec6beb1_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110325738 took 72 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 72 ms
for event 7ec6beb1_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 7ec6beb1_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ec6beb1_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 7ec6beb1_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7ec6beb1_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7ec6beb1_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 7ec6beb1_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ec6beb1_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7ec6beb1_TASK
END: Invoking TASK controller pipeline for event MessageChange::7ec6beb1_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 3d57188a_TASK. Event session ID: 3000a812cfc0036
Event 3d57188a_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 3d57188a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 3d57188a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 3d57188a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::3d57188a_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::3d57188a_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 60 ms for event 7ec6beb1_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 7ec6beb1_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7ec6beb1_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::3d57188a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::3d57188a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325870, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7ec6beb1_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Global baseline calculation completed and has been persisted into metadata store.
Event PinotCluster::TASK::3d57188a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::3d57188a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110325896, took 21 ms
Event 3d57188a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110325843 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event 3d57188a_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 3d57188a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3d57188a_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
93 ms for event 7ec6beb1_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 3d57188a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 3d57188a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 3d57188a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 3d57188a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3d57188a_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3d57188a_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::3d57188a_TASK
for cluster PinotCluster, took 147 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 7ec6beb1_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 7ec6beb1_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 7ec6beb1_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 7ec6beb1_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ec6beb1_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 7ec6beb1_DEFAULT : Sending Message 4999ac3a-30d1-4046-ba52-0d938ae12410 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__19__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 7ec6beb1_DEFAULT : Sending Message f3aebde4-0100-4918-970a-cbdfc3b066f4 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__19__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 7ec6beb1_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7ec6beb1_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7ec6beb1_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::7ec6beb1_DEFAULT
for cluster PinotCluster, took 308 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 206 ms
TotalProcessed time for event: MessageChange took: 308 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 3d57188a_DEFAULT. Event session ID: 3000a812cfc0036
Event 3d57188a_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 3d57188a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 3d57188a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 3d57188a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f48e0927_TASK. Event session ID: 3000a812cfc0036
Event f48e0927_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f48e0927_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f48e0927_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f48e0927_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
Event PinotCluster::TASK::f48e0927_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f48e0927_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326073, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 3d57188a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3d57188a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326103, took 0 ms
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 4ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::f48e0927_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::f48e0927_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326076, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event 3d57188a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326046 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event 3d57188a_DEFAULT
Event PinotCluster::TASK::f48e0927_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f48e0927_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326108, took 20 ms
Event f48e0927_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326049 took 80 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 80 ms
for event f48e0927_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 3d57188a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 3d57188a_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event f48e0927_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f48e0927_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 74 ms for event 3d57188a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 3d57188a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3d57188a_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 18 ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
76 ms for event f48e0927_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f48e0927_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f48e0927_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f48e0927_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f48e0927_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f48e0927_TASK
END: Invoking TASK controller pipeline for event MessageChange::f48e0927_TASK for
cluster PinotCluster, took 175 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 549aea7b_TASK. Event session ID: 3000a812cfc0036
Event 549aea7b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 549aea7b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 549aea7b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 549aea7b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::549aea7b_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 3d57188a_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::549aea7b_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 3d57188a_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 3d57188a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3d57188a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 3d57188a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 3d57188a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 3d57188a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3d57188a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3d57188a_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::3d57188a_DEFAULT for cluster PinotCluster, took 235 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 464 ms
TotalProcessed time for event: IdealStateChange took: 235 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ede7bc0b_DEFAULT. Event session ID: 3000a812cfc0036
Event ede7bc0b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event ede7bc0b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ede7bc0b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ede7bc0b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::549aea7b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::549aea7b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326256, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::549aea7b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::549aea7b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326282, took 20 ms
Event 549aea7b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326224 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 549aea7b_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event 549aea7b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 549aea7b_TASK
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326306, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ede7bc0b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ede7bc0b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326330, took 0 ms
Event ede7bc0b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326281 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event ede7bc0b_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event ede7bc0b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ede7bc0b_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 549aea7b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 549aea7b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 549aea7b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 549aea7b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 549aea7b_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 549aea7b_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::549aea7b_TASK
for cluster PinotCluster, took 155 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ede7bc0b_TASK. Event session ID: 3000a812cfc0036
Event ede7bc0b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event ede7bc0b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ede7bc0b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ede7bc0b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::ede7bc0b_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ede7bc0b_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event ede7bc0b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ede7bc0b_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ede7bc0b_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::ede7bc0b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ede7bc0b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326403, took 35 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event ede7bc0b_DEFAULT
Event PinotCluster::TASK::ede7bc0b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ede7bc0b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110326438, took 20 ms
Event ede7bc0b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326379 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event ede7bc0b_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event ede7bc0b_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event ede7bc0b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ede7bc0b_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event ede7bc0b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ede7bc0b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event ede7bc0b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ede7bc0b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event ede7bc0b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ede7bc0b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ede7bc0b_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::ede7bc0b_DEFAULT
for cluster PinotCluster, took 213 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 173 ms
TotalProcessed time for event: MessageChange took: 213 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 549aea7b_DEFAULT. Event session ID: 3000a812cfc0036
Event 549aea7b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 549aea7b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 549aea7b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 549aea7b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event ede7bc0b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ede7bc0b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ede7bc0b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ede7bc0b_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ede7bc0b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ede7bc0b_TASK
END: Invoking TASK controller pipeline for event MessageChange::ede7bc0b_TASK for
cluster PinotCluster, took 153 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326519, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 549aea7b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::549aea7b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110326543, took 0 ms
Event 549aea7b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110326494 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 549aea7b_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 549aea7b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 549aea7b_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 549aea7b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 549aea7b_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 549aea7b_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 549aea7b_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 549aea7b_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 549aea7b_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 60 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 549aea7b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 549aea7b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 549aea7b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 549aea7b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 549aea7b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 549aea7b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 549aea7b_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::549aea7b_DEFAULT for cluster PinotCluster, took 211 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 393 ms
TotalProcessed time for event: CurrentStateChange took: 211 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9b057c31_DEFAULT. Event session ID: 3000a812cfc0036
Event 9b057c31_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9b057c31_TASK. Event session ID: 3000a812cfc0036
Event 9b057c31_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 9b057c31_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 9b057c31_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 9b057c31_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 9b057c31_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 9b057c31_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 9b057c31_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::9b057c31_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::9b057c31_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/65834060-c30f-472a-a9da-
0e6d38259e0d=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/5f2e71e7-f3a4-442b-bb98-
3f5001dc0168=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/04ad0dd6-553e-4b9f-98b9-
0195e65a5b49=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/65834060-c30f-472a-a9da-
0e6d38259e0d=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/5f2e71e7-f3a4-442b-bb98-
3f5001dc0168=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/04ad0dd6-553e-4b9f-98b9-
0195e65a5b49=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328071, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 9b057c31_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::9b057c31_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328095, took 0 ms
Event PinotCluster::TASK::9b057c31_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9b057c31_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328071, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 9b057c31_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328045 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 9b057c31_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Event PinotCluster::TASK::9b057c31_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9b057c31_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328099, took 20 ms
Event 9b057c31_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328045 took 75 for TASK pipeline
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event 9b057c31_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 9b057c31_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 9b057c31_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 17
ms for event 9b057c31_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 9b057c31_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 52 ms for event 9b057c31_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 9b057c31_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9b057c31_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 9b057c31_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 9b057c31_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9b057c31_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 9b057c31_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9b057c31_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9b057c31_TASK
END: Invoking TASK controller pipeline for event MessageChange::9b057c31_TASK for
cluster PinotCluster, took 145 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: fad59de7_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event 9b057c31_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::fad59de7_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::fad59de7_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event fad59de7_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event fad59de7_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event fad59de7_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::fad59de7_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::fad59de7_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 9b057c31_DEFAULT
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/65468cb4-7da5-4ac8-946d-
6daf0c8664c0=-101}
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 9b057c31_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9b057c31_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 9b057c31_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 9b057c31_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 9b057c31_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 9b057c31_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9b057c31_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::9b057c31_DEFAULT
for cluster PinotCluster, took 201 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 201 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 58a11ee9_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::fad59de7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fad59de7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328231, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event 58a11ee9_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 58a11ee9_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 58a11ee9_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::fad59de7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fad59de7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328255, took 20 ms
Event fad59de7_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328190 took 86 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event fad59de7_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event fad59de7_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fad59de7_TASK
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328284, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 58a11ee9_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::58a11ee9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328308, took 0 ms
Event 58a11ee9_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328246 took 67 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 67 ms
for event 58a11ee9_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 58a11ee9_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 58a11ee9_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event fad59de7_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fad59de7_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fad59de7_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event fad59de7_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fad59de7_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fad59de7_TASK
END: Invoking TASK controller pipeline for event MessageChange::fad59de7_TASK for
cluster PinotCluster, took 158 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e48f084c_TASK. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::e48f084c_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::e48f084c_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event e48f084c_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e48f084c_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e48f084c_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::e48f084c_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e48f084c_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 58a11ee9_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 58a11ee9_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 58a11ee9_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 58a11ee9_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::e48f084c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e48f084c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328388, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::e48f084c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e48f084c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328412, took 19 ms
Event e48f084c_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328348 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event e48f084c_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event e48f084c_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e48f084c_TASK
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
84 ms for event 58a11ee9_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event 58a11ee9_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event e48f084c_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e48f084c_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e48f084c_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e48f084c_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e48f084c_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e48f084c_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::e48f084c_TASK
for cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 58a11ee9_TASK. Event session ID: 3000a812cfc0036
Event 58a11ee9_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 58a11ee9_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 58a11ee9_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 58a11ee9_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::58a11ee9_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::58a11ee9_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 58a11ee9_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 58a11ee9_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 58a11ee9_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 58a11ee9_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 58a11ee9_DEFAULT : Sending Message cb440783-63c5-4331-9706-e0c0b0d406f2 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__5__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 58a11ee9_DEFAULT : Sending Message fb1d684e-ebe2-41dd-a287-91808a03c726 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__5__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event 58a11ee9_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 58a11ee9_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 58a11ee9_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::58a11ee9_DEFAULT
for cluster PinotCluster, took 282 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 21 ms
TotalProcessed time for event: MessageChange took: 282 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e48f084c_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event e48f084c_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e48f084c_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e48f084c_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::58a11ee9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::58a11ee9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328533, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 27
ms.
Start to refresh stale message cache
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::58a11ee9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::58a11ee9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328558, took 25 ms
Event 58a11ee9_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328505 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 58a11ee9_TASK
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 58a11ee9_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 58a11ee9_TASK
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328575, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e48f084c_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e48f084c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328604, took 0 ms
Event e48f084c_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328528 took 81 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 81 ms
for event e48f084c_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event e48f084c_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event e48f084c_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 58a11ee9_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 58a11ee9_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 58a11ee9_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 58a11ee9_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 58a11ee9_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 58a11ee9_TASK
END: Invoking TASK controller pipeline for event MessageChange::58a11ee9_TASK for
cluster PinotCluster, took 152 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: dedd91ea_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 1 ms
Event dedd91ea_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event dedd91ea_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event dedd91ea_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event dedd91ea_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::dedd91ea_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::dedd91ea_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event e48f084c_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e48f084c_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e48f084c_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event e48f084c_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::dedd91ea_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::dedd91ea_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328682, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::dedd91ea_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::dedd91ea_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328710, took 19 ms
Computing BestPossibleMapping for clickstreamd_REALTIME
Event dedd91ea_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328658 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event dedd91ea_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event dedd91ea_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event dedd91ea_TASK
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
85 ms for event e48f084c_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event e48f084c_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event dedd91ea_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event dedd91ea_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event dedd91ea_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event dedd91ea_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event dedd91ea_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event dedd91ea_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::dedd91ea_TASK for
cluster PinotCluster, took 145 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: be28f35c_TASK. Event session ID: 3000a812cfc0036
Event be28f35c_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event be28f35c_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event be28f35c_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event be28f35c_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::be28f35c_TASK : Refreshed 9 property StateModelDefinition
took 17 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::be28f35c_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 27
ms for event e48f084c_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e48f084c_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event e48f084c_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e48f084c_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event e48f084c_DEFAULT : Sending Message 13277297-d41c-46cb-a10c-231ea4f21232 to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__22__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event e48f084c_DEFAULT : Sending Message 4e01fa00-3bee-44a3-9464-352569f3829f to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__22__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event e48f084c_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e48f084c_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e48f084c_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::e48f084c_DEFAULT for cluster PinotCluster, took 314 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 256 ms
TotalProcessed time for event: IdealStateChange took: 314 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: dedd91ea_DEFAULT. Event session ID: 3000a812cfc0036
Event dedd91ea_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event dedd91ea_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event dedd91ea_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event dedd91ea_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
Event PinotCluster::TASK::be28f35c_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::be28f35c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328848, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::be28f35c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::be28f35c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328875, took 25 ms
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328869, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event dedd91ea_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::dedd91ea_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110328900, took 0 ms
Event be28f35c_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328803 took 98 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 98 ms
for event be28f35c_TASK
Event dedd91ea_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328843 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event dedd91ea_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event be28f35c_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be28f35c_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event dedd91ea_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event dedd91ea_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event be28f35c_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event be28f35c_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event be28f35c_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event be28f35c_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be28f35c_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be28f35c_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::be28f35c_TASK
for cluster PinotCluster, took 169 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 23cf1f2e_TASK. Event session ID: 3000a812cfc0036
Event 23cf1f2e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 23cf1f2e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 23cf1f2e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 23cf1f2e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::23cf1f2e_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::23cf1f2e_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event dedd91ea_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event dedd91ea_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event dedd91ea_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event dedd91ea_DEFAULT
Event PinotCluster::TASK::23cf1f2e_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::23cf1f2e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110328996, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event dedd91ea_DEFAULT
Event PinotCluster::TASK::23cf1f2e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::23cf1f2e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329024, took 20 ms
Event 23cf1f2e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110328972 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event 23cf1f2e_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event dedd91ea_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event dedd91ea_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event dedd91ea_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event dedd91ea_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event dedd91ea_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event dedd91ea_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event dedd91ea_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::dedd91ea_DEFAULT
for cluster PinotCluster, took 215 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 258 ms
TotalProcessed time for event: MessageChange took: 215 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: da727265_DEFAULT. Event session ID: 3000a812cfc0036
Event da727265_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event da727265_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event da727265_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event da727265_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::da727265_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::da727265_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 23cf1f2e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 23cf1f2e_TASK
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::da727265_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::da727265_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329083, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event da727265_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::da727265_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::da727265_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329107, took 0 ms
Event da727265_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329058 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event da727265_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 23cf1f2e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 23cf1f2e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 23cf1f2e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 23cf1f2e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 23cf1f2e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 23cf1f2e_TASK
END: Invoking TASK controller pipeline for event MessageChange::23cf1f2e_TASK for
cluster PinotCluster, took 145 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: da727265_TASK. Event session ID: 3000a812cfc0036
Event da727265_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event da727265_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event da727265_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event da727265_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::da727265_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::da727265_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event da727265_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event da727265_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::da727265_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::da727265_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329142, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::da727265_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::da727265_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329165, took 20 ms
Event da727265_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329117 took 68 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 68 ms
for event da727265_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event da727265_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event da727265_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event da727265_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event da727265_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event da727265_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event da727265_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event da727265_DEFAULT
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 48
ms for event da727265_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 72 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event da727265_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event da727265_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
69 ms for event da727265_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event da727265_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event da727265_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event da727265_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event da727265_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event da727265_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::da727265_TASK
for cluster PinotCluster, took 157 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event da727265_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event da727265_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event da727265_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event da727265_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event da727265_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::da727265_DEFAULT for cluster PinotCluster, took 224 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 170 ms
TotalProcessed time for event: CurrentStateChange took: 224 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 23cf1f2e_DEFAULT. Event session ID: 3000a812cfc0036
Event 23cf1f2e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 23cf1f2e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 23cf1f2e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 23cf1f2e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329307, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 23cf1f2e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::23cf1f2e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329331, took 0 ms
Event 23cf1f2e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329282 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 23cf1f2e_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 23cf1f2e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 23cf1f2e_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 23cf1f2e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 23cf1f2e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 23cf1f2e_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 23cf1f2e_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 23cf1f2e_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 23cf1f2e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 23cf1f2e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 23cf1f2e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 23cf1f2e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 23cf1f2e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 23cf1f2e_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 23cf1f2e_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::23cf1f2e_DEFAULT
for cluster PinotCluster, took 210 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 390 ms
TotalProcessed time for event: MessageChange took: 210 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b68acb64_DEFAULT. Event session ID: 3000a812cfc0036
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event b68acb64_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event b68acb64_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b68acb64_TASK. Event session ID: 3000a812cfc0036
Event b68acb64_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event b68acb64_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event b68acb64_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event b68acb64_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b68acb64_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b68acb64_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::b68acb64_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b68acb64_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/a157535c-f95d-4452-ad20-
9c7de900e24b=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/04ecd71f-b408-427c-95d5-
0a834d1b1367=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ec32baa0-8e42-4b1e-8bce-
ec7a24b60bc3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/a157535c-f95d-4452-ad20-
9c7de900e24b=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/04ecd71f-b408-427c-95d5-
0a834d1b1367=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ec32baa0-8e42-4b1e-8bce-
ec7a24b60bc3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329569, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b68acb64_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b68acb64_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329593, took 0 ms
Event PinotCluster::TASK::b68acb64_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b68acb64_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329569, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b68acb64_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329542 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event b68acb64_DEFAULT
Event PinotCluster::TASK::b68acb64_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b68acb64_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329596, took 20 ms
Event b68acb64_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329542 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event b68acb64_TASK
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event b68acb64_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event b68acb64_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event b68acb64_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b68acb64_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event b68acb64_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event b68acb64_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b68acb64_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event b68acb64_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b68acb64_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b68acb64_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event b68acb64_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b68acb64_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b68acb64_TASK
END: Invoking TASK controller pipeline for event MessageChange::b68acb64_TASK for
cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 5d1971d9_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event b68acb64_DEFAULT
Event PinotCluster::TASK::5d1971d9_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::5d1971d9_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event 5d1971d9_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 5d1971d9_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 5d1971d9_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::5d1971d9_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::5d1971d9_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event b68acb64_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event b68acb64_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b68acb64_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event b68acb64_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b68acb64_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 1 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event b68acb64_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b68acb64_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b68acb64_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::b68acb64_DEFAULT
for cluster PinotCluster, took 211 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 211 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a59d7c87_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::5d1971d9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::5d1971d9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329729, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event a59d7c87_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event a59d7c87_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event a59d7c87_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::5d1971d9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::5d1971d9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329754, took 20 ms
Event 5d1971d9_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329690 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 85 ms
for event 5d1971d9_TASK
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 5d1971d9_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 5d1971d9_TASK
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329792, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event a59d7c87_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a59d7c87_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110329815, took 1 ms
Event a59d7c87_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329753 took 67 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 67 ms
for event a59d7c87_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event a59d7c87_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event a59d7c87_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 5d1971d9_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 5d1971d9_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 5d1971d9_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 5d1971d9_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 5d1971d9_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 5d1971d9_TASK
END: Invoking TASK controller pipeline for event MessageChange::5d1971d9_TASK for
cluster PinotCluster, took 158 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: c6cb4173_TASK. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::c6cb4173_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::c6cb4173_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event c6cb4173_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c6cb4173_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c6cb4173_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::c6cb4173_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c6cb4173_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event a59d7c87_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event a59d7c87_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a59d7c87_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event a59d7c87_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::c6cb4173_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c6cb4173_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329887, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::c6cb4173_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c6cb4173_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110329911, took 19 ms
Event c6cb4173_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110329848 took 83 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 83 ms
for event c6cb4173_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 36
ms for event c6cb4173_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6cb4173_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
100 ms for event a59d7c87_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event c6cb4173_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c6cb4173_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c6cb4173_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event c6cb4173_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6cb4173_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6cb4173_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::c6cb4173_TASK
for cluster PinotCluster, took 176 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a59d7c87_TASK. Event session ID: 3000a812cfc0036
Event a59d7c87_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event a59d7c87_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event a59d7c87_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event a59d7c87_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event PinotCluster::TASK::a59d7c87_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event a59d7c87_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::a59d7c87_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event a59d7c87_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event a59d7c87_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event a59d7c87_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a59d7c87_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event a59d7c87_DEFAULT : Sending Message 32a28421-f4d8-4166-a6c3-896af404bbb5 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__33__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event a59d7c87_DEFAULT : Sending Message 2531f695-174e-40e3-86db-8b315f20a9b4 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__33__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event a59d7c87_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a59d7c87_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a59d7c87_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::a59d7c87_DEFAULT
for cluster PinotCluster, took 302 ms
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 13 ms
TotalProcessed time for event: MessageChange took: 302 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: c6cb4173_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event c6cb4173_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event c6cb4173_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event c6cb4173_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::a59d7c87_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a59d7c87_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330050, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : Refreshed 1 property
ClusterConstraint took 5 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::a59d7c87_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a59d7c87_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330078, took 21 ms
Event a59d7c87_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330024 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event a59d7c87_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event a59d7c87_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a59d7c87_TASK
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330101, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c6cb4173_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6cb4173_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330130, took 0 ms
Event c6cb4173_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330056 took 79 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 79 ms
for event c6cb4173_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event c6cb4173_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c6cb4173_DEFAULT
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event a59d7c87_TASK
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a59d7c87_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a59d7c87_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event a59d7c87_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a59d7c87_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a59d7c87_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::a59d7c87_TASK for
cluster PinotCluster, took 153 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b73008b5_TASK. Event session ID: 3000a812cfc0036
Event b73008b5_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event b73008b5_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b73008b5_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b73008b5_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::b73008b5_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b73008b5_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 58 ms for event c6cb4173_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event c6cb4173_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6cb4173_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event c6cb4173_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Event PinotCluster::TASK::b73008b5_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::b73008b5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330202, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Event PinotCluster::TASK::b73008b5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b73008b5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330229, took 20 ms
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event b73008b5_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330177 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event b73008b5_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event b73008b5_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event b73008b5_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
96 ms for event c6cb4173_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event b73008b5_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b73008b5_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b73008b5_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event b73008b5_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b73008b5_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b73008b5_TASK
END: Invoking TASK controller pipeline for event MessageChange::b73008b5_TASK for
cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: d132be97_TASK. Event session ID: 3000a812cfc0036
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event PinotCluster::TASK::d132be97_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::d132be97_TASK : Refreshed 11 property IdealState took 18
ms. Selective: true
Event d132be97_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event d132be97_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event d132be97_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::d132be97_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 35
ms for event c6cb4173_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::d132be97_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event c6cb4173_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6cb4173_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 8 ms
for event c6cb4173_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c6cb4173_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event c6cb4173_DEFAULT : Sending Message 9482a165-e4f7-4cf5-8460-c72730df8dc8 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__13__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event c6cb4173_DEFAULT : Sending Message 19d36a8c-a144-48bd-99ce-2635d46f7818 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__13__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event c6cb4173_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c6cb4173_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6cb4173_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::c6cb4173_DEFAULT for cluster PinotCluster, took 319 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 266 ms
TotalProcessed time for event: IdealStateChange took: 319 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 11459511_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 2
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::11459511_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::11459511_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event 11459511_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 11459511_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 11459511_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::d132be97_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d132be97_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330368, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::11459511_DEFAULT : Refreshed 9 property
StateModelDefinition took 6 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::11459511_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::d132be97_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d132be97_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330396, took 21 ms
Event d132be97_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330325 took 93 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 93 ms
for event d132be97_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event d132be97_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d132be97_TASK
Event PinotCluster::DEFAULT::11459511_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::11459511_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330421, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 11459511_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::11459511_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::11459511_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330449, took 0 ms
Event 11459511_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330375 took 79 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 79 ms
for event 11459511_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 11459511_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 11459511_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
57 ms for event d132be97_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event d132be97_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d132be97_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event d132be97_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d132be97_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d132be97_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::d132be97_TASK
for cluster PinotCluster, took 172 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ec8f6658_TASK. Event session ID: 3000a812cfc0036
Event ec8f6658_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event ec8f6658_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ec8f6658_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ec8f6658_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::ec8f6658_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ec8f6658_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 11459511_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 11459511_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 11459511_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 11459511_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Computing BestPossibleMapping for amxordersntt_REALTIME
Event PinotCluster::TASK::ec8f6658_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::ec8f6658_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330522, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::ec8f6658_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ec8f6658_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330550, took 20 ms
Event ec8f6658_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330497 took 74 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 74 ms
for event ec8f6658_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event ec8f6658_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event ec8f6658_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
88 ms for event 11459511_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event ec8f6658_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event ec8f6658_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ec8f6658_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ec8f6658_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ec8f6658_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ec8f6658_TASK
END: Invoking TASK controller pipeline for event MessageChange::ec8f6658_TASK for
cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: c88c87c8_TASK. Event session ID: 3000a812cfc0036
Event c88c87c8_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event c88c87c8_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c88c87c8_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c88c87c8_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::c88c87c8_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c88c87c8_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 11459511_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 11459511_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 11459511_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 11459511_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 11459511_DEFAULT
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 11459511_DEFAULT : Sending Message 3e4bc655-dc97-4a2e-b07c-6e7aaea96764 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__28__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 11459511_DEFAULT : Sending Message bdcae242-37ef-4c9a-9947-64d672c35d35 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__28__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 10 ms for event 11459511_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 11459511_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 11459511_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::11459511_DEFAULT
for cluster PinotCluster, took 306 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 194 ms
TotalProcessed time for event: MessageChange took: 306 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: ae13d08e_DEFAULT. Event session ID: 3000a812cfc0036
Event ae13d08e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event ae13d08e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ae13d08e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ae13d08e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::c88c87c8_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c88c87c8_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330671, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 0ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::c88c87c8_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c88c87c8_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330699, took 22 ms
Event c88c87c8_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330645 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event c88c87c8_TASK
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 5ms
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330709, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ae13d08e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ae13d08e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330739, took 0 ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event c88c87c8_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event c88c87c8_TASK
Event ae13d08e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330681 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event ae13d08e_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event ae13d08e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ae13d08e_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event c88c87c8_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c88c87c8_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c88c87c8_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event c88c87c8_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c88c87c8_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c88c87c8_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::c88c87c8_TASK
for cluster PinotCluster, took 151 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 52256b2b_TASK. Event session ID: 3000a812cfc0036
Event 52256b2b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 52256b2b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 52256b2b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 52256b2b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::52256b2b_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::52256b2b_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event ae13d08e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ae13d08e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ae13d08e_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ae13d08e_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 32
ms.
Start to refresh stale message cache
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event ae13d08e_DEFAULT
Event PinotCluster::TASK::52256b2b_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::52256b2b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330834, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event ae13d08e_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 63 ms
Event PinotCluster::TASK::52256b2b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::52256b2b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330863, took 25 ms
Event 52256b2b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330796 took 93 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 93 ms
for event 52256b2b_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event ae13d08e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ae13d08e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event ae13d08e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ae13d08e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event ae13d08e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ae13d08e_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ae13d08e_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::ae13d08e_DEFAULT for cluster PinotCluster, took 223 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 255 ms
TotalProcessed time for event: CurrentStateChange took: 223 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: c88c87c8_DEFAULT. Event session ID: 3000a812cfc0036
Event c88c87c8_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event c88c87c8_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event c88c87c8_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event c88c87c8_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 52256b2b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 52256b2b_TASK
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330930, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c88c87c8_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c88c87c8_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110330954, took 0 ms
Event c88c87c8_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330904 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event c88c87c8_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 52256b2b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 52256b2b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 52256b2b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 52256b2b_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52256b2b_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52256b2b_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::52256b2b_TASK
for cluster PinotCluster, took 167 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
Controller stopping periodical rebalance timer at period 9223372036854775807
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 056d68bc_TASK. Event session ID: 3000a812cfc0036
Event 056d68bc_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 056d68bc_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 056d68bc_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 056d68bc_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::056d68bc_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::056d68bc_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event c88c87c8_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c88c87c8_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 26
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::056d68bc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::056d68bc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110330995, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event c88c87c8_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event c88c87c8_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c88c87c8_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::056d68bc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::056d68bc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110331020, took 33 ms
Event 056d68bc_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110330963 took 91 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 91 ms
for event 056d68bc_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event c88c87c8_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 056d68bc_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 056d68bc_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event c88c87c8_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event c88c87c8_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event c88c87c8_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event c88c87c8_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c88c87c8_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event c88c87c8_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c88c87c8_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c88c87c8_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::c88c87c8_DEFAULT for cluster PinotCluster, took 216 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 676 ms
TotalProcessed time for event: IdealStateChange took: 216 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 056d68bc_DEFAULT. Event session ID: 3000a812cfc0036
Event 056d68bc_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Event 056d68bc_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 056d68bc_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 056d68bc_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 056d68bc_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 056d68bc_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 056d68bc_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 056d68bc_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 056d68bc_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 056d68bc_TASK
END: Invoking TASK controller pipeline for event MessageChange::056d68bc_TASK for
cluster PinotCluster, took 165 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110331146, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 056d68bc_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::056d68bc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110331172, took 0 ms
Event 056d68bc_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110331120 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event 056d68bc_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 056d68bc_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 056d68bc_DEFAULT
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 71 ms for event 056d68bc_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 056d68bc_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 056d68bc_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 056d68bc_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event 056d68bc_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 056d68bc_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 056d68bc_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 5 ms
for event 056d68bc_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 056d68bc_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 056d68bc_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 056d68bc_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 056d68bc_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::056d68bc_DEFAULT
for cluster PinotCluster, took 224 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 389 ms
TotalProcessed time for event: MessageChange took: 224 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 52256b2b_DEFAULT. Event session ID: 3000a812cfc0036
Event 52256b2b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 52256b2b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 52256b2b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 52256b2b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110331370, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 52256b2b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52256b2b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110331394, took 0 ms
Event 52256b2b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110331344 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 52256b2b_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 52256b2b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 52256b2b_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 52256b2b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 52256b2b_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
5 ms for event 52256b2b_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52256b2b_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 52256b2b_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 47 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 52256b2b_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 52256b2b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52256b2b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 52256b2b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52256b2b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 52256b2b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 52256b2b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52256b2b_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::52256b2b_DEFAULT for cluster PinotCluster, took 219 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 617 ms
TotalProcessed time for event: CurrentStateChange took: 219 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 700f47a6_DEFAULT. Event session ID: 3000a812cfc0036
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Event 700f47a6_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 700f47a6_TASK. Event session ID: 3000a812cfc0036
Event 700f47a6_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 700f47a6_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 700f47a6_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 700f47a6_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 700f47a6_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 700f47a6_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 700f47a6_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::700f47a6_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::700f47a6_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ed9a26fe-6d50-4106-bf40-
a9aa77cfc576=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/534de9b7-126a-4362-9dc0-
89cb5af563be=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/4a336633-1881-4979-a2b2-
3c42c88748e3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ed9a26fe-6d50-4106-bf40-
a9aa77cfc576=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/534de9b7-126a-4362-9dc0-
89cb5af563be=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/4a336633-1881-4979-a2b2-
3c42c88748e3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332185, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 700f47a6_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::700f47a6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332210, took 1 ms
Event PinotCluster::TASK::700f47a6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::700f47a6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332186, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 700f47a6_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332159 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event 700f47a6_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::700f47a6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::700f47a6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332214, took 20 ms
Event 700f47a6_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332159 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event 700f47a6_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 700f47a6_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 700f47a6_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 700f47a6_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 700f47a6_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 700f47a6_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 700f47a6_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 700f47a6_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event 700f47a6_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 700f47a6_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 700f47a6_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 700f47a6_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 700f47a6_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 700f47a6_TASK
END: Invoking TASK controller pipeline for event MessageChange::700f47a6_TASK for
cluster PinotCluster, took 152 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 957b6c3d_TASK. Event session ID: 3000a812cfc0036
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
31 ms for event 700f47a6_DEFAULT
Event PinotCluster::TASK::957b6c3d_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::957b6c3d_TASK : Refreshed 11 property IdealState took 16
ms. Selective: true
Event 957b6c3d_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 957b6c3d_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 957b6c3d_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::957b6c3d_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::957b6c3d_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 700f47a6_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 700f47a6_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 700f47a6_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 700f47a6_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 700f47a6_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::957b6c3d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::957b6c3d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332352, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 700f47a6_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 700f47a6_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 700f47a6_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::700f47a6_DEFAULT
for cluster PinotCluster, took 221 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 221 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 957b6c3d_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event 957b6c3d_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 957b6c3d_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 957b6c3d_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::957b6c3d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::957b6c3d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332376, took 21 ms
Event 957b6c3d_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332311 took 87 for TASK pipeline
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event 957b6c3d_TASK
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 22
ms for event 957b6c3d_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 957b6c3d_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332421, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 957b6c3d_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::957b6c3d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332446, took 0 ms
Event 957b6c3d_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332380 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 71 ms
for event 957b6c3d_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 957b6c3d_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 957b6c3d_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
57 ms for event 957b6c3d_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 957b6c3d_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 957b6c3d_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 957b6c3d_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 957b6c3d_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 957b6c3d_TASK
END: Invoking TASK controller pipeline for event MessageChange::957b6c3d_TASK for
cluster PinotCluster, took 167 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 52fee39e_TASK. Event session ID: 3000a812cfc0036
Event 52fee39e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 52fee39e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 52fee39e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 52fee39e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::52fee39e_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::52fee39e_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 957b6c3d_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 957b6c3d_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 957b6c3d_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::52fee39e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::52fee39e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332503, took 34 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 957b6c3d_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
Event PinotCluster::TASK::52fee39e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::52fee39e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332537, took 21 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Event 52fee39e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332478 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event 52fee39e_TASK
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 52fee39e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52fee39e_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
90 ms for event 957b6c3d_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 52fee39e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 52fee39e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 52fee39e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event 52fee39e_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52fee39e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 52fee39e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::52fee39e_TASK
for cluster PinotCluster, took 157 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 957b6c3d_DEFAULT
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 957b6c3d_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 957b6c3d_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 957b6c3d_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 957b6c3d_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 957b6c3d_DEFAULT : Sending Message efefe409-ed9d-4e35-a5ba-27eb0f1a3376 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__6__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 957b6c3d_DEFAULT : Sending Message 4b09a178-2095-4c88-986b-9d35debd730d to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__6__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event 957b6c3d_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 957b6c3d_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 957b6c3d_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::957b6c3d_DEFAULT
for cluster PinotCluster, took 296 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 195 ms
TotalProcessed time for event: MessageChange took: 296 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 52fee39e_DEFAULT. Event session ID: 3000a812cfc0036
Event 52fee39e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 52fee39e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 52fee39e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 52fee39e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 2
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 3ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bef9a056_TASK. Event session ID: 3000a812cfc0036
Event bef9a056_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event bef9a056_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event bef9a056_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event bef9a056_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::bef9a056_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::bef9a056_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 4ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332705, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 52fee39e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::52fee39e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332734, took 0 ms
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::bef9a056_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::bef9a056_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332709, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 52fee39e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332676 took 65 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 65 ms
for event 52fee39e_DEFAULT
Event PinotCluster::TASK::bef9a056_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bef9a056_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332741, took 20 ms
Event bef9a056_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332680 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event bef9a056_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 52fee39e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52fee39e_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event bef9a056_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event bef9a056_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 52fee39e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 52fee39e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52fee39e_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event bef9a056_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bef9a056_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bef9a056_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event bef9a056_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bef9a056_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bef9a056_TASK
END: Invoking TASK controller pipeline for event MessageChange::bef9a056_TASK for
cluster PinotCluster, took 156 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 36961fb0_TASK. Event session ID: 3000a812cfc0036
Event 36961fb0_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 36961fb0_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 36961fb0_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 36961fb0_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::36961fb0_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::36961fb0_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 52fee39e_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 52fee39e_DEFAULT
Event PinotCluster::TASK::36961fb0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::36961fb0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332861, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 52fee39e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52fee39e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 52fee39e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52fee39e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 52fee39e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 52fee39e_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 52fee39e_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::52fee39e_DEFAULT for cluster PinotCluster, took 225 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 445 ms
TotalProcessed time for event: IdealStateChange took: 225 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bcb0e1d9_DEFAULT. Event session ID: 3000a812cfc0036
Event bcb0e1d9_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event bcb0e1d9_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event bcb0e1d9_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event bcb0e1d9_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::36961fb0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::36961fb0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110332885, took 22 ms
Event 36961fb0_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332836 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event 36961fb0_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event 36961fb0_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36961fb0_TASK
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332926, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event bcb0e1d9_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bcb0e1d9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110332950, took 0 ms
Event bcb0e1d9_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332901 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event bcb0e1d9_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event bcb0e1d9_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bcb0e1d9_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 36961fb0_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 36961fb0_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 36961fb0_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 36961fb0_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36961fb0_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36961fb0_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::36961fb0_TASK
for cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bcb0e1d9_TASK. Event session ID: 3000a812cfc0036
Event bcb0e1d9_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event bcb0e1d9_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event bcb0e1d9_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event bcb0e1d9_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::bcb0e1d9_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::bcb0e1d9_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 65 ms for event bcb0e1d9_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event bcb0e1d9_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bcb0e1d9_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::bcb0e1d9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bcb0e1d9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110333009, took 33 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event bcb0e1d9_DEFAULT
Event PinotCluster::TASK::bcb0e1d9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bcb0e1d9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110333042, took 27 ms
Event bcb0e1d9_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110332984 took 86 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event bcb0e1d9_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event bcb0e1d9_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bcb0e1d9_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event bcb0e1d9_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event bcb0e1d9_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event bcb0e1d9_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event bcb0e1d9_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bcb0e1d9_DEFAULT
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event bcb0e1d9_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bcb0e1d9_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bcb0e1d9_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::bcb0e1d9_DEFAULT
for cluster PinotCluster, took 220 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 168 ms
TotalProcessed time for event: MessageChange took: 220 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 36961fb0_DEFAULT. Event session ID: 3000a812cfc0036
Event 36961fb0_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 36961fb0_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 36961fb0_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 36961fb0_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event bcb0e1d9_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bcb0e1d9_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bcb0e1d9_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event bcb0e1d9_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bcb0e1d9_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bcb0e1d9_TASK
END: Invoking TASK controller pipeline for event MessageChange::bcb0e1d9_TASK for
cluster PinotCluster, took 157 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110333145, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 36961fb0_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::36961fb0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110333170, took 0 ms
Event 36961fb0_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110333121 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 36961fb0_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 36961fb0_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36961fb0_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 36961fb0_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 36961fb0_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 36961fb0_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36961fb0_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 36961fb0_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 36961fb0_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 64 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 36961fb0_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 36961fb0_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 36961fb0_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36961fb0_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 36961fb0_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36961fb0_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 36961fb0_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::36961fb0_DEFAULT for cluster PinotCluster, took 208 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 394 ms
TotalProcessed time for event: CurrentStateChange took: 208 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: edb83dfd_DEFAULT. Event session ID: 3000a812cfc0036
Event edb83dfd_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event edb83dfd_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: edb83dfd_TASK. Event session ID: 3000a812cfc0036
Event edb83dfd_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event edb83dfd_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event edb83dfd_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event edb83dfd_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event edb83dfd_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event edb83dfd_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::edb83dfd_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::edb83dfd_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
START: InstanceMessagesCache.refresh()
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/e8cb75ef-246e-46e3-b6a3-
1a543571159c=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/f630651b-3ba0-4570-a9ae-
209d58b68e1f=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/98ef0fda-8a68-4bd5-8fd6-
0b4620cced62=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/e8cb75ef-246e-46e3-b6a3-
1a543571159c=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/f630651b-3ba0-4570-a9ae-
209d58b68e1f=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/98ef0fda-8a68-4bd5-8fd6-
0b4620cced62=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::edb83dfd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::edb83dfd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335597, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110335597, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event edb83dfd_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::edb83dfd_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110335626, took 0 ms
Event edb83dfd_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110335570 took 62 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 62 ms
for event edb83dfd_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Event PinotCluster::TASK::edb83dfd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::edb83dfd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335622, took 24 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event edb83dfd_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110335570 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event edb83dfd_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event edb83dfd_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event edb83dfd_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event edb83dfd_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event edb83dfd_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event edb83dfd_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event edb83dfd_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event edb83dfd_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event edb83dfd_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event edb83dfd_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event edb83dfd_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event edb83dfd_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event edb83dfd_TASK
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event edb83dfd_TASK
END: Invoking TASK controller pipeline for event MessageChange::edb83dfd_TASK for
cluster PinotCluster, took 149 ms
Finish calculating the new best possible assignment.
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: c6902882_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event edb83dfd_DEFAULT
Event PinotCluster::TASK::c6902882_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::c6902882_TASK : Refreshed 11 property IdealState took 16
ms. Selective: true
Event c6902882_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c6902882_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c6902882_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::c6902882_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c6902882_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event edb83dfd_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event edb83dfd_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event edb83dfd_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event edb83dfd_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event edb83dfd_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event edb83dfd_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event edb83dfd_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event edb83dfd_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::edb83dfd_DEFAULT
for cluster PinotCluster, took 213 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 213 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: c6902882_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::c6902882_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c6902882_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335759, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::c6902882_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6902882_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event c6902882_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event c6902882_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event c6902882_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::c6902882_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::c6902882_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::c6902882_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c6902882_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335783, took 22 ms
Event c6902882_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110335719 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event c6902882_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event c6902882_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6902882_TASK
Event PinotCluster::DEFAULT::c6902882_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6902882_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110335824, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c6902882_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::c6902882_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c6902882_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110335848, took 0 ms
Event c6902882_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110335783 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 70 ms
for event c6902882_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event c6902882_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event c6902882_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event c6902882_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c6902882_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c6902882_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event c6902882_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6902882_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c6902882_TASK
END: Invoking TASK controller pipeline for event MessageChange::c6902882_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 421b9566_TASK. Event session ID: 3000a812cfc0036
Event 421b9566_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 421b9566_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 421b9566_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 421b9566_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::421b9566_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::421b9566_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event c6902882_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event c6902882_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6902882_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::421b9566_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::421b9566_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335906, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c6902882_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::421b9566_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::421b9566_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110335931, took 20 ms
Event 421b9566_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110335880 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event 421b9566_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 44
ms for event 421b9566_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 421b9566_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
109 ms for event c6902882_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event 421b9566_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 421b9566_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 421b9566_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 421b9566_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 421b9566_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 421b9566_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::421b9566_TASK
for cluster PinotCluster, took 173 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event c6902882_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event c6902882_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6902882_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event c6902882_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c6902882_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event c6902882_DEFAULT : Sending Message 69ce28dd-c7f3-4008-a288-f8e5054f54a9 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__24__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event c6902882_DEFAULT : Sending Message 637cd937-ad9a-4b7c-93f7-e855cf00517f to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__24__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event c6902882_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c6902882_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c6902882_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::c6902882_DEFAULT
for cluster PinotCluster, took 316 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 188 ms
TotalProcessed time for event: MessageChange took: 316 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 421b9566_DEFAULT. Event session ID: 3000a812cfc0036
Event 421b9566_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 421b9566_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 421b9566_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 421b9566_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::421b9566_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ab86ff4f_TASK. Event session ID: 3000a812cfc0036
Event ab86ff4f_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event ab86ff4f_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ab86ff4f_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ab86ff4f_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::421b9566_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
Event PinotCluster::TASK::ab86ff4f_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ab86ff4f_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::421b9566_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::421b9566_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336126, took 33 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 421b9566_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::421b9566_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::421b9566_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336159, took 0 ms
Event 421b9566_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336099 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event 421b9566_DEFAULT
Event PinotCluster::TASK::ab86ff4f_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::ab86ff4f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336129, took 36 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 421b9566_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 421b9566_DEFAULT
Event PinotCluster::TASK::ab86ff4f_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ab86ff4f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336165, took 22 ms
Event ab86ff4f_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336102 took 86 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event ab86ff4f_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event ab86ff4f_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ab86ff4f_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 421b9566_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 421b9566_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 421b9566_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event ab86ff4f_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ab86ff4f_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ab86ff4f_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ab86ff4f_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ab86ff4f_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ab86ff4f_TASK
END: Invoking TASK controller pipeline for event MessageChange::ab86ff4f_TASK for
cluster PinotCluster, took 162 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 3b8cf2f5_TASK. Event session ID: 3000a812cfc0036
Event 3b8cf2f5_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 3b8cf2f5_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 3b8cf2f5_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 3b8cf2f5_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::3b8cf2f5_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 421b9566_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::3b8cf2f5_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 35
ms for event 421b9566_DEFAULT
Event PinotCluster::TASK::3b8cf2f5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::3b8cf2f5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336289, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 421b9566_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 421b9566_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 421b9566_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 421b9566_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 421b9566_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 421b9566_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 421b9566_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::421b9566_DEFAULT for cluster PinotCluster, took 228 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 455 ms
TotalProcessed time for event: IdealStateChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a37b4272_DEFAULT. Event session ID: 3000a812cfc0036
Event a37b4272_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event a37b4272_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event a37b4272_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event a37b4272_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::3b8cf2f5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::3b8cf2f5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336314, took 22 ms
Event 3b8cf2f5_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336264 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event 3b8cf2f5_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 3b8cf2f5_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 3b8cf2f5_TASK
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336353, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event a37b4272_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a37b4272_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336378, took 0 ms
Event a37b4272_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336327 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event a37b4272_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event a37b4272_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event a37b4272_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 3b8cf2f5_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 3b8cf2f5_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 3b8cf2f5_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 3b8cf2f5_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3b8cf2f5_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3b8cf2f5_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::3b8cf2f5_TASK
for cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a37b4272_TASK. Event session ID: 3000a812cfc0036
Event a37b4272_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event a37b4272_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event a37b4272_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event a37b4272_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::a37b4272_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::a37b4272_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event a37b4272_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event a37b4272_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a37b4272_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::a37b4272_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a37b4272_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336437, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event a37b4272_DEFAULT
Event PinotCluster::TASK::a37b4272_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a37b4272_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336468, took 20 ms
Event a37b4272_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336412 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event a37b4272_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event a37b4272_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a37b4272_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event a37b4272_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event a37b4272_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event a37b4272_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event a37b4272_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a37b4272_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event a37b4272_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a37b4272_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a37b4272_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::a37b4272_DEFAULT
for cluster PinotCluster, took 216 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 171 ms
TotalProcessed time for event: MessageChange took: 216 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 3b8cf2f5_DEFAULT. Event session ID: 3000a812cfc0036
Event 3b8cf2f5_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 3b8cf2f5_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 3b8cf2f5_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 3b8cf2f5_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event a37b4272_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a37b4272_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a37b4272_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event a37b4272_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a37b4272_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a37b4272_TASK
END: Invoking TASK controller pipeline for event MessageChange::a37b4272_TASK for
cluster PinotCluster, took 151 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336571, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 3b8cf2f5_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3b8cf2f5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336599, took 0 ms
Event 3b8cf2f5_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336543 took 62 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 62 ms
for event 3b8cf2f5_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 3b8cf2f5_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3b8cf2f5_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 3b8cf2f5_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 3b8cf2f5_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3b8cf2f5_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3b8cf2f5_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 24 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 3b8cf2f5_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 3b8cf2f5_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 62 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 3b8cf2f5_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3b8cf2f5_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 987bd446_TASK. Event session ID: 3000a812cfc0036
Event 987bd446_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 987bd446_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 987bd446_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 987bd446_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 3b8cf2f5_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3b8cf2f5_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::987bd446_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::987bd446_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 3b8cf2f5_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 3b8cf2f5_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3b8cf2f5_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::3b8cf2f5_DEFAULT for cluster PinotCluster, took 221 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 395 ms
TotalProcessed time for event: CurrentStateChange took: 221 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 987bd446_DEFAULT. Event session ID: 3000a812cfc0036
Event 987bd446_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 987bd446_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 987bd446_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 987bd446_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
Event PinotCluster::DEFAULT::987bd446_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::987bd446_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/1d89e330-3995-4370-8ab3-
d4198e66fb4d=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/6540efd1-874e-496f-aa2d-
cc8c1219b91b=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/0e70fad8-e8fd-4cc0-8c57-
933c660e5143=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/6540efd1-874e-496f-aa2d-
cc8c1219b91b=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/0e70fad8-e8fd-4cc0-8c57-
933c660e5143=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::987bd446_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::987bd446_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336787, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::987bd446_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::987bd446_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336792, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 987bd446_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::987bd446_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::987bd446_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110336820, took 0 ms
Event 987bd446_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336764 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event 987bd446_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::987bd446_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::987bd446_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336813, took 26 ms
Event 987bd446_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336759 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event 987bd446_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 987bd446_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 987bd446_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 987bd446_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 987bd446_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 987bd446_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 987bd446_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 987bd446_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 987bd446_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 987bd446_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 987bd446_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 987bd446_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 987bd446_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 987bd446_TASK
END: Invoking TASK controller pipeline for event MessageChange::987bd446_TASK for
cluster PinotCluster, took 156 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 73aa9386_TASK. Event session ID: 3000a812cfc0036
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
26 ms for event 987bd446_DEFAULT
Event PinotCluster::TASK::73aa9386_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::73aa9386_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 73aa9386_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 73aa9386_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 73aa9386_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::73aa9386_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::73aa9386_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 36
ms for event 987bd446_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 987bd446_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 987bd446_DEFAULT
Event PinotCluster::TASK::73aa9386_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::73aa9386_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336956, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 987bd446_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 987bd446_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 987bd446_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 987bd446_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 987bd446_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::987bd446_DEFAULT
for cluster PinotCluster, took 226 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 6 ms
TotalProcessed time for event: MessageChange took: 226 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 73aa9386_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::73aa9386_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::73aa9386_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110336981, took 20 ms
Event 73aa9386_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336915 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event 73aa9386_TASK
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event 73aa9386_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 73aa9386_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 73aa9386_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 73aa9386_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 73aa9386_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337030, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 73aa9386_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::73aa9386_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337055, took 0 ms
Event 73aa9386_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110336990 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 70 ms
for event 73aa9386_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 73aa9386_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 73aa9386_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 73aa9386_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 73aa9386_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 73aa9386_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 73aa9386_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END: Invoking TASK controller pipeline for event MessageChange::73aa9386_TASK for
cluster PinotCluster, took 160 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: d218c6dc_TASK. Event session ID: 3000a812cfc0036
Event d218c6dc_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event d218c6dc_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event d218c6dc_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event d218c6dc_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::d218c6dc_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::d218c6dc_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 73aa9386_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 73aa9386_DEFAULT
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::d218c6dc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d218c6dc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337100, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 73aa9386_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 73aa9386_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 73aa9386_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event 73aa9386_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Event PinotCluster::TASK::d218c6dc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d218c6dc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337124, took 29 ms
Event d218c6dc_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337076 took 78 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 78 ms
for event d218c6dc_TASK
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event d218c6dc_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d218c6dc_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
89 ms for event 73aa9386_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event d218c6dc_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d218c6dc_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d218c6dc_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event d218c6dc_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d218c6dc_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d218c6dc_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::d218c6dc_TASK
for cluster PinotCluster, took 152 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 73aa9386_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 73aa9386_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 73aa9386_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 73aa9386_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 73aa9386_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 73aa9386_DEFAULT : Sending Message 6ed93c6b-bdb3-4962-a4c2-306421ff87b5 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__0__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 73aa9386_DEFAULT : Sending Message 24a4f6aa-f98a-4199-83c9-03f91f44a3c8 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__0__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 73aa9386_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 73aa9386_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 73aa9386_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::73aa9386_DEFAULT
for cluster PinotCluster, took 295 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 204 ms
TotalProcessed time for event: MessageChange took: 295 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: d218c6dc_DEFAULT. Event session ID: 3000a812cfc0036
Event d218c6dc_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event d218c6dc_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event d218c6dc_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event d218c6dc_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: fcb2b87b_TASK. Event session ID: 3000a812cfc0036
Event fcb2b87b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event fcb2b87b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event fcb2b87b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event fcb2b87b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
Event PinotCluster::TASK::fcb2b87b_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::fcb2b87b_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 6ms
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337312, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event d218c6dc_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d218c6dc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337344, took 0 ms
Event PinotCluster::TASK::fcb2b87b_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::fcb2b87b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337315, took 34 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event d218c6dc_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337285 took 65 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 65 ms
for event d218c6dc_DEFAULT
Event PinotCluster::TASK::fcb2b87b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fcb2b87b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337350, took 19 ms
Event fcb2b87b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337288 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event fcb2b87b_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event d218c6dc_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event d218c6dc_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event fcb2b87b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event fcb2b87b_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event d218c6dc_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event d218c6dc_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d218c6dc_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event fcb2b87b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fcb2b87b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fcb2b87b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event fcb2b87b_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fcb2b87b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fcb2b87b_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::fcb2b87b_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: bba1c1a0_TASK. Event session ID: 3000a812cfc0036
Event bba1c1a0_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event bba1c1a0_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event bba1c1a0_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event bba1c1a0_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::bba1c1a0_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::bba1c1a0_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event d218c6dc_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 34
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event d218c6dc_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event d218c6dc_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event d218c6dc_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event d218c6dc_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d218c6dc_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::bba1c1a0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bba1c1a0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337485, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event d218c6dc_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d218c6dc_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d218c6dc_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::d218c6dc_DEFAULT for cluster PinotCluster, took 225 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 454 ms
TotalProcessed time for event: IdealStateChange took: 225 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 86a345ce_DEFAULT. Event session ID: 3000a812cfc0036
Event 86a345ce_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 86a345ce_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 86a345ce_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 86a345ce_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::bba1c1a0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bba1c1a0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337510, took 20 ms
Event bba1c1a0_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337445 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event bba1c1a0_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event bba1c1a0_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bba1c1a0_TASK
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337535, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 86a345ce_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::86a345ce_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337560, took 0 ms
Event 86a345ce_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337510 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 86a345ce_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 86a345ce_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86a345ce_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event bba1c1a0_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 17 ms for
event bba1c1a0_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bba1c1a0_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event bba1c1a0_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bba1c1a0_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bba1c1a0_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::bba1c1a0_TASK
for cluster PinotCluster, took 177 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 86a345ce_TASK. Event session ID: 3000a812cfc0036
Event 86a345ce_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 86a345ce_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 86a345ce_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 86a345ce_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::86a345ce_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::86a345ce_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 73 ms for event 86a345ce_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 86a345ce_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 86a345ce_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Event PinotCluster::TASK::86a345ce_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::86a345ce_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337647, took 37 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 86a345ce_DEFAULT
Event PinotCluster::TASK::86a345ce_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::86a345ce_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110337684, took 20 ms
Event 86a345ce_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337622 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event 86a345ce_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 86a345ce_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 86a345ce_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 86a345ce_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 86a345ce_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 86a345ce_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 86a345ce_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86a345ce_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 3 ms for event 86a345ce_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86a345ce_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 86a345ce_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::86a345ce_DEFAULT
for cluster PinotCluster, took 228 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 175 ms
TotalProcessed time for event: MessageChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: bba1c1a0_DEFAULT. Event session ID: 3000a812cfc0036
Event bba1c1a0_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event bba1c1a0_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event bba1c1a0_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event bba1c1a0_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 86a345ce_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 86a345ce_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 86a345ce_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 86a345ce_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 86a345ce_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 86a345ce_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::86a345ce_TASK for
cluster PinotCluster, took 158 ms
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337763, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event bba1c1a0_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::bba1c1a0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110337787, took 0 ms
Event bba1c1a0_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110337738 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event bba1c1a0_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event bba1c1a0_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bba1c1a0_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event bba1c1a0_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event bba1c1a0_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event bba1c1a0_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bba1c1a0_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event bba1c1a0_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event bba1c1a0_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 64 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event bba1c1a0_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bba1c1a0_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 8 ms
for event bba1c1a0_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bba1c1a0_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event bba1c1a0_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event bba1c1a0_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event bba1c1a0_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::bba1c1a0_DEFAULT for cluster PinotCluster, took 217 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 1 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 407 ms
TotalProcessed time for event: CurrentStateChange took: 217 ms
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ad9ac6ff_DEFAULT. Event session ID: 3000a812cfc0036
Event ad9ac6ff_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event ad9ac6ff_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ad9ac6ff_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ad9ac6ff_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ad9ac6ff_TASK. Event session ID: 3000a812cfc0036
Event ad9ac6ff_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event ad9ac6ff_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ad9ac6ff_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ad9ac6ff_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::ad9ac6ff_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ad9ac6ff_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/0973f96a-38a5-4c31-a694-
03ae973a3698=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e503c7fb-2c67-4cda-8c58-
0bc446cbd3df=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/0973f96a-38a5-4c31-a694-
03ae973a3698=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e503c7fb-2c67-4cda-8c58-
0bc446cbd3df=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/593a9329-d9b6-41d7-bc86-
d1d10ccf30de=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::ad9ac6ff_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ad9ac6ff_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343312, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343312, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ad9ac6ff_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ad9ac6ff_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343340, took 0 ms
Event ad9ac6ff_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343283 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event ad9ac6ff_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::ad9ac6ff_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ad9ac6ff_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343336, took 22 ms
Event ad9ac6ff_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343283 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event ad9ac6ff_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event ad9ac6ff_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ad9ac6ff_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event ad9ac6ff_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event ad9ac6ff_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event ad9ac6ff_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ad9ac6ff_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ad9ac6ff_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
58 ms for event ad9ac6ff_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ad9ac6ff_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ad9ac6ff_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ad9ac6ff_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ad9ac6ff_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ad9ac6ff_TASK
END: Invoking TASK controller pipeline for event MessageChange::ad9ac6ff_TASK for
cluster PinotCluster, took 156 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 2a94a228_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event ad9ac6ff_DEFAULT
Event PinotCluster::TASK::2a94a228_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::2a94a228_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 2a94a228_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 2a94a228_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 2a94a228_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::2a94a228_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::2a94a228_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event ad9ac6ff_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event ad9ac6ff_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ad9ac6ff_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event ad9ac6ff_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ad9ac6ff_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event ad9ac6ff_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ad9ac6ff_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ad9ac6ff_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::ad9ac6ff_DEFAULT
for cluster PinotCluster, took 214 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 214 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 2a94a228_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::2a94a228_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2a94a228_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343479, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : Refreshed 11 property IdealState
took 17 ms. Selective: true
Event 2a94a228_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 2a94a228_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 2a94a228_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::2a94a228_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2a94a228_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343503, took 21 ms
Event 2a94a228_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343439 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 85 ms
for event 2a94a228_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 2a94a228_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2a94a228_TASK
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343539, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 2a94a228_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2a94a228_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343563, took 0 ms
Event 2a94a228_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343497 took 71 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 71 ms
for event 2a94a228_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 2a94a228_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2a94a228_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 2a94a228_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2a94a228_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2a94a228_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 2a94a228_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2a94a228_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2a94a228_TASK
END: Invoking TASK controller pipeline for event MessageChange::2a94a228_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 4b02b51b_TASK. Event session ID: 3000a812cfc0036
Event 4b02b51b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 4b02b51b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 4b02b51b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 4b02b51b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::4b02b51b_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::4b02b51b_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 2a94a228_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 2a94a228_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2a94a228_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::4b02b51b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4b02b51b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343621, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 2a94a228_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::4b02b51b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4b02b51b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343646, took 20 ms
Event 4b02b51b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343596 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 4b02b51b_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 4b02b51b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4b02b51b_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
82 ms for event 2a94a228_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 4b02b51b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4b02b51b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4b02b51b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 4b02b51b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4b02b51b_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4b02b51b_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::4b02b51b_TASK
for cluster PinotCluster, took 142 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 2a94a228_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 2a94a228_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2a94a228_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 2a94a228_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2a94a228_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 2a94a228_DEFAULT : Sending Message 56724fbf-47d7-406b-8f65-bf7708fa8f26 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__3__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 2a94a228_DEFAULT : Sending Message 76e8ea57-1d79-49e4-a6c5-a2c102d131ff to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__3__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 7 ms for event 2a94a228_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2a94a228_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2a94a228_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::2a94a228_DEFAULT
for cluster PinotCluster, took 280 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 189 ms
TotalProcessed time for event: MessageChange took: 280 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 4b02b51b_DEFAULT. Event session ID: 3000a812cfc0036
Event 4b02b51b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 4b02b51b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 4b02b51b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 4b02b51b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 47b5e9d2_TASK. Event session ID: 3000a812cfc0036
Event 47b5e9d2_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 47b5e9d2_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 47b5e9d2_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 47b5e9d2_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::47b5e9d2_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::47b5e9d2_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343805, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 4b02b51b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4b02b51b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110343832, took 0 ms
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 4
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 5ms
Event 4b02b51b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343777 took 61 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 61 ms
for event 4b02b51b_DEFAULT
Event PinotCluster::TASK::47b5e9d2_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::47b5e9d2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343808, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 4b02b51b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 4b02b51b_DEFAULT
Event PinotCluster::TASK::47b5e9d2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::47b5e9d2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343839, took 19 ms
Event 47b5e9d2_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343780 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 47b5e9d2_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 47b5e9d2_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 47b5e9d2_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 4b02b51b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 4b02b51b_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4b02b51b_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 47b5e9d2_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 47b5e9d2_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 47b5e9d2_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 47b5e9d2_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 47b5e9d2_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 47b5e9d2_TASK
END: Invoking TASK controller pipeline for event MessageChange::47b5e9d2_TASK for
cluster PinotCluster, took 148 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 34f49eee_TASK. Event session ID: 3000a812cfc0036
Event 34f49eee_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 34f49eee_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 34f49eee_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 34f49eee_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::34f49eee_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::34f49eee_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 4b02b51b_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 4b02b51b_DEFAULT
Event PinotCluster::TASK::34f49eee_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::34f49eee_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343952, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 4b02b51b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 4b02b51b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 4b02b51b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4b02b51b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 4b02b51b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4b02b51b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4b02b51b_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::4b02b51b_DEFAULT for cluster PinotCluster, took 213 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 422 ms
TotalProcessed time for event: IdealStateChange took: 213 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 4c75f9d3_DEFAULT. Event session ID: 3000a812cfc0036
Event 4c75f9d3_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 4c75f9d3_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 4c75f9d3_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 4c75f9d3_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::34f49eee_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::34f49eee_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110343977, took 21 ms
Event 34f49eee_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343928 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 34f49eee_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 34f49eee_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 34f49eee_TASK
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344017, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 4c75f9d3_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4c75f9d3_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344042, took 0 ms
Event 4c75f9d3_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110343991 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event 4c75f9d3_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 4c75f9d3_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4c75f9d3_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 34f49eee_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 34f49eee_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 34f49eee_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 34f49eee_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 34f49eee_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 34f49eee_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event CurrentStateChange::34f49eee_TASK
for cluster PinotCluster, took 144 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 4c75f9d3_TASK. Event session ID: 3000a812cfc0036
Event 4c75f9d3_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 4c75f9d3_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 4c75f9d3_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 4c75f9d3_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::4c75f9d3_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::4c75f9d3_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::4c75f9d3_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4c75f9d3_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344098, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 4c75f9d3_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 4c75f9d3_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4c75f9d3_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::4c75f9d3_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4c75f9d3_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344122, took 20 ms
Event 4c75f9d3_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344072 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 4c75f9d3_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 4c75f9d3_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 4c75f9d3_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4c75f9d3_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 4c75f9d3_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 4c75f9d3_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 4c75f9d3_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 4c75f9d3_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4c75f9d3_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 4c75f9d3_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4c75f9d3_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4c75f9d3_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::4c75f9d3_DEFAULT
for cluster PinotCluster, took 232 ms
Callback time for event: MessageChange took: 4 ms
InQueue time for event: MessageChange took: 155 ms
TotalProcessed time for event: MessageChange took: 232 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START AsyncProcess: DEFAULT::PersistAssignmentStage
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 34f49eee_DEFAULT. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Event 34f49eee_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 34f49eee_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 34f49eee_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 34f49eee_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
71 ms for event 4c75f9d3_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4c75f9d3_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4c75f9d3_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event 4c75f9d3_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4c75f9d3_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4c75f9d3_TASK
END: Invoking TASK controller pipeline for event MessageChange::4c75f9d3_TASK for
cluster PinotCluster, took 161 ms
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344248, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 34f49eee_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::34f49eee_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344272, took 0 ms
Event 34f49eee_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344223 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 34f49eee_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 34f49eee_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 34f49eee_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 52 ms for event 34f49eee_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 34f49eee_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 34f49eee_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 34f49eee_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 34f49eee_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 34f49eee_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 55 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 34f49eee_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 34f49eee_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 34f49eee_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 34f49eee_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 34f49eee_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 34f49eee_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 34f49eee_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::34f49eee_DEFAULT for cluster PinotCluster, took 204 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 393 ms
TotalProcessed time for event: CurrentStateChange took: 204 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f02bcc90_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f02bcc90_TASK. Event session ID: 3000a812cfc0036
Event f02bcc90_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f02bcc90_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f02bcc90_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f02bcc90_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event f02bcc90_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event f02bcc90_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event f02bcc90_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event f02bcc90_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::f02bcc90_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f02bcc90_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
START: InstanceMessagesCache.refresh()
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ad5bda9b-7ea1-43a5-b70a-
c686a67ae67e=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/da94e725-30fe-4cfc-aed2-
35d8d6d12f4a=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/4c8db251-a04f-408b-873a-
1051cc2bdd3e=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/ad5bda9b-7ea1-43a5-b70a-
c686a67ae67e=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/da94e725-30fe-4cfc-aed2-
35d8d6d12f4a=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/4c8db251-a04f-408b-873a-
1051cc2bdd3e=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::f02bcc90_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f02bcc90_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344747, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344748, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f02bcc90_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f02bcc90_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344775, took 0 ms
Event f02bcc90_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344719 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event f02bcc90_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Event PinotCluster::TASK::f02bcc90_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f02bcc90_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344771, took 23 ms
Event f02bcc90_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344719 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event f02bcc90_TASK
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event f02bcc90_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event f02bcc90_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event f02bcc90_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f02bcc90_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 52 ms for event f02bcc90_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event f02bcc90_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f02bcc90_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event f02bcc90_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f02bcc90_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f02bcc90_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f02bcc90_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f02bcc90_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f02bcc90_TASK
END: Invoking TASK controller pipeline for event MessageChange::f02bcc90_TASK for
cluster PinotCluster, took 149 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1db91b93_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 24 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event f02bcc90_DEFAULT
Event PinotCluster::TASK::1db91b93_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::1db91b93_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 1db91b93_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 1db91b93_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 1db91b93_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::1db91b93_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::1db91b93_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 36
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 48
ms for event f02bcc90_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event f02bcc90_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f02bcc90_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event f02bcc90_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f02bcc90_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::1db91b93_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1db91b93_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344924, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event f02bcc90_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f02bcc90_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f02bcc90_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::f02bcc90_DEFAULT
for cluster PinotCluster, took 230 ms
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 230 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1db91b93_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event 1db91b93_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 1db91b93_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 1db91b93_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::1db91b93_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1db91b93_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110344948, took 21 ms
Event 1db91b93_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344868 took 102 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 102 ms
for event 1db91b93_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 1db91b93_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 1db91b93_TASK
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110344988, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 1db91b93_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1db91b93_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345012, took 0 ms
Event 1db91b93_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110344949 took 68 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 68 ms
for event 1db91b93_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 1db91b93_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1db91b93_DEFAULT
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 1db91b93_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1db91b93_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 1db91b93_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 1db91b93_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1db91b93_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1db91b93_TASK
END: Invoking TASK controller pipeline for event MessageChange::1db91b93_TASK for
cluster PinotCluster, took 174 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 872e413f_TASK. Event session ID: 3000a812cfc0036
Event 872e413f_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 872e413f_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 872e413f_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 872e413f_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::872e413f_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::872e413f_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 1db91b93_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 1db91b93_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1db91b93_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::872e413f_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::872e413f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345066, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 1db91b93_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::872e413f_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::872e413f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345090, took 21 ms
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event 872e413f_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345042 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 872e413f_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 872e413f_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 872e413f_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
94 ms for event 1db91b93_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
67 ms for event 872e413f_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 872e413f_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 872e413f_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 872e413f_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 872e413f_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 872e413f_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::872e413f_TASK
for cluster PinotCluster, took 155 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 61773fb9_TASK. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::61773fb9_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::61773fb9_TASK : Refreshed 11 property IdealState took 17
ms. Selective: true
Event 61773fb9_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 61773fb9_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 61773fb9_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 1db91b93_DEFAULT
Event PinotCluster::TASK::61773fb9_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::61773fb9_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 1db91b93_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1db91b93_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 1db91b93_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1db91b93_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 1db91b93_DEFAULT : Sending Message ac7cee65-faa6-48d8-8322-519819de0b4f to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__34__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 1db91b93_DEFAULT : Sending Message efe86599-da36-4fa9-b28c-06d97ea6ff98 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__34__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 1db91b93_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1db91b93_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1db91b93_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::1db91b93_DEFAULT
for cluster PinotCluster, took 294 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 204 ms
TotalProcessed time for event: MessageChange took: 294 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: b0141a66_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 28
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event b0141a66_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event b0141a66_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event b0141a66_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
Event PinotCluster::TASK::61773fb9_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::61773fb9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345248, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::61773fb9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::61773fb9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345277, took 20 ms
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event 61773fb9_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345197 took 101 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 101 ms
for event 61773fb9_TASK
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345288, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b0141a66_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b0141a66_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345316, took 0 ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event 61773fb9_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 61773fb9_TASK
Event b0141a66_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345243 took 78 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 78 ms
for event b0141a66_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event b0141a66_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event b0141a66_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 61773fb9_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 61773fb9_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 61773fb9_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 61773fb9_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 61773fb9_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 61773fb9_TASK
END: Invoking TASK controller pipeline for event MessageChange::61773fb9_TASK for
cluster PinotCluster, took 177 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: b0141a66_TASK. Event session ID: 3000a812cfc0036
Event b0141a66_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event b0141a66_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b0141a66_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b0141a66_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::b0141a66_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b0141a66_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event b0141a66_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event b0141a66_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b0141a66_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event b0141a66_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
Event PinotCluster::TASK::b0141a66_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::b0141a66_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345399, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
Event PinotCluster::TASK::b0141a66_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b0141a66_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345427, took 20 ms
Event b0141a66_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345374 took 74 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 74 ms
for event b0141a66_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event b0141a66_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event b0141a66_TASK
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
88 ms for event b0141a66_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event b0141a66_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event b0141a66_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b0141a66_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b0141a66_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event b0141a66_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b0141a66_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b0141a66_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::b0141a66_TASK
for cluster PinotCluster, took 150 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: be733c0f_TASK. Event session ID: 3000a812cfc0036
Event be733c0f_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event be733c0f_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event be733c0f_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event be733c0f_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::be733c0f_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::be733c0f_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event b0141a66_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b0141a66_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event b0141a66_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b0141a66_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event b0141a66_DEFAULT : Sending Message 1869e772-d822-4f29-b4c6-e8453cf8327d to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__15__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event b0141a66_DEFAULT : Sending Message c17cbf12-bbf9-453d-874a-8f65a70f3d16 to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__15__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event b0141a66_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b0141a66_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b0141a66_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::b0141a66_DEFAULT for cluster PinotCluster, took 302 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 137 ms
TotalProcessed time for event: IdealStateChange took: 302 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: be733c0f_DEFAULT. Event session ID: 3000a812cfc0036
Event be733c0f_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event be733c0f_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event be733c0f_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event be733c0f_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
Event PinotCluster::TASK::be733c0f_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::be733c0f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345550, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345570, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event be733c0f_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::be733c0f_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345602, took 0 ms
Event PinotCluster::TASK::be733c0f_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::be733c0f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345579, took 24 ms
Event be733c0f_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345524 took 80 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 80 ms
for event be733c0f_TASK
Event be733c0f_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345545 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event be733c0f_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event be733c0f_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be733c0f_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event be733c0f_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event be733c0f_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event be733c0f_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event be733c0f_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event be733c0f_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event be733c0f_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be733c0f_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event be733c0f_TASK
END: Invoking TASK controller pipeline for event MessageChange::be733c0f_TASK for
cluster PinotCluster, took 155 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 4995d2cc_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event 4995d2cc_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 4995d2cc_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 4995d2cc_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 4995d2cc_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::4995d2cc_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::4995d2cc_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event be733c0f_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event be733c0f_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event be733c0f_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
START: InstanceMessagesCache.refresh()
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event be733c0f_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event be733c0f_DEFAULT
Event PinotCluster::TASK::4995d2cc_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::4995d2cc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345715, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event be733c0f_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event be733c0f_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event be733c0f_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event be733c0f_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event be733c0f_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event be733c0f_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event be733c0f_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::be733c0f_DEFAULT
for cluster PinotCluster, took 218 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 248 ms
TotalProcessed time for event: MessageChange took: 218 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 4995d2cc_DEFAULT. Event session ID: 3000a812cfc0036
Event 4995d2cc_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 4995d2cc_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 4995d2cc_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 4995d2cc_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::4995d2cc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4995d2cc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345743, took 20 ms
Event 4995d2cc_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345679 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 85 ms
for event 4995d2cc_TASK
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 4995d2cc_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4995d2cc_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345788, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 4995d2cc_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4995d2cc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110345812, took 0 ms
Event 4995d2cc_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345763 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 4995d2cc_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 4995d2cc_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4995d2cc_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4995d2cc_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 4995d2cc_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4995d2cc_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4995d2cc_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::4995d2cc_TASK
for cluster PinotCluster, took 156 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f5f03787_TASK. Event session ID: 3000a812cfc0036
Event f5f03787_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f5f03787_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f5f03787_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f5f03787_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::f5f03787_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 4995d2cc_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4995d2cc_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f5f03787_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::f5f03787_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f5f03787_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345860, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 4995d2cc_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 4995d2cc_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4995d2cc_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4995d2cc_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::f5f03787_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f5f03787_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110345885, took 31 ms
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event f5f03787_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345835 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event f5f03787_TASK
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
28 ms for event 4995d2cc_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event f5f03787_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f5f03787_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 4995d2cc_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 70 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 4995d2cc_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 4995d2cc_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 5 ms
for event 4995d2cc_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4995d2cc_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 4995d2cc_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4995d2cc_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4995d2cc_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::4995d2cc_DEFAULT for cluster PinotCluster, took 213 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 171 ms
TotalProcessed time for event: CurrentStateChange took: 213 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f5f03787_DEFAULT. Event session ID: 3000a812cfc0036
Event f5f03787_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event f5f03787_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event f5f03787_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event f5f03787_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event f5f03787_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f5f03787_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f5f03787_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f5f03787_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f5f03787_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f5f03787_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END: Invoking TASK controller pipeline for event MessageChange::f5f03787_TASK for
cluster PinotCluster, took 155 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110346002, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f5f03787_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f5f03787_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110346028, took 0 ms
Event f5f03787_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110345976 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event f5f03787_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event f5f03787_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f5f03787_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event f5f03787_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event f5f03787_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f5f03787_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event f5f03787_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event f5f03787_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event f5f03787_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event f5f03787_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event f5f03787_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f5f03787_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event f5f03787_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f5f03787_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f5f03787_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::f5f03787_DEFAULT
for cluster PinotCluster, took 214 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 376 ms
TotalProcessed time for event: MessageChange took: 214 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 36546651_TASK. Event session ID: 3000a812cfc0036
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 36546651_DEFAULT. Event session ID: 3000a812cfc0036
Event 36546651_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 36546651_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 36546651_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 36546651_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 36546651_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 36546651_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 36546651_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event 36546651_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::36546651_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
Event PinotCluster::DEFAULT::36546651_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::36546651_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::36546651_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/5c2c2fc4-63c6-4bbf-a6d2-
3e49321e0f52=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/89ada112-8201-441f-bb05-
9cdd56f50303=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/bcbe8df3-1a92-41c3-b7dc-
2e2b507eaff1=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/5c2c2fc4-63c6-4bbf-a6d2-
3e49321e0f52=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/89ada112-8201-441f-bb05-
9cdd56f50303=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/bcbe8df3-1a92-41c3-b7dc-
2e2b507eaff1=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::36546651_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::36546651_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110346856, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::36546651_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::36546651_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110346857, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 36546651_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::36546651_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::36546651_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110346886, took 0 ms
Event 36546651_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110346828 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event 36546651_DEFAULT
Event PinotCluster::TASK::36546651_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::36546651_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110346882, took 22 ms
Event 36546651_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110346828 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event 36546651_TASK
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 36546651_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36546651_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 36546651_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36546651_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 36546651_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 36546651_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 36546651_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 36546651_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 36546651_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 36546651_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 36546651_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36546651_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 36546651_TASK
END: Invoking TASK controller pipeline for event MessageChange::36546651_TASK for
cluster PinotCluster, took 150 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 57421831_TASK. Event session ID: 3000a812cfc0036
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 36546651_DEFAULT
Event PinotCluster::TASK::57421831_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::57421831_TASK : Refreshed 11 property IdealState took 16
ms. Selective: true
Event 57421831_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 57421831_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 57421831_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::57421831_TASK : Refreshed 9 property StateModelDefinition
took 16 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::57421831_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 36546651_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 36546651_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 36546651_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 36546651_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36546651_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 36546651_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 36546651_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 36546651_DEFAULT
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::36546651_DEFAULT
for cluster PinotCluster, took 220 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 0 ms
TotalProcessed time for event: MessageChange took: 220 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 57421831_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::57421831_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::57421831_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347032, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::57421831_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::57421831_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event 57421831_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 57421831_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 57421831_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::57421831_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::57421831_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::57421831_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::57421831_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347058, took 21 ms
Event 57421831_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110346978 took 102 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 102 ms
for event 57421831_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 57421831_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 57421831_TASK
Event PinotCluster::DEFAULT::57421831_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::57421831_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347090, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 57421831_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::57421831_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::57421831_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347115, took 0 ms
Event 57421831_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347048 took 72 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 72 ms
for event 57421831_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 57421831_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 57421831_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 57421831_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 57421831_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 57421831_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 57421831_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 57421831_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 57421831_TASK
END: Invoking TASK controller pipeline for event MessageChange::57421831_TASK for
cluster PinotCluster, took 177 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: cffe0e03_TASK. Event session ID: 3000a812cfc0036
Event cffe0e03_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event cffe0e03_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event cffe0e03_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event cffe0e03_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::cffe0e03_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::cffe0e03_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 57421831_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 57421831_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 57421831_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::cffe0e03_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::cffe0e03_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347180, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 57421831_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::cffe0e03_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::cffe0e03_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347208, took 19 ms
Event cffe0e03_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347155 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event cffe0e03_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 34 ms
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event cffe0e03_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event cffe0e03_TASK
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event cffe0e03_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event cffe0e03_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event cffe0e03_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event cffe0e03_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event cffe0e03_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event cffe0e03_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::cffe0e03_TASK
for cluster PinotCluster, took 148 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
108 ms for event 57421831_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 57421831_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 57421831_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 57421831_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 57421831_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 57421831_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 57421831_DEFAULT : Sending Message 511d07c4-5ccf-4893-bd95-862c6aa0aa69 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__11__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 57421831_DEFAULT : Sending Message b59d20ba-1710-4435-bc4d-a6b6eeae3cd5 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__11__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 10 ms for event 57421831_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 57421831_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 57421831_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::57421831_DEFAULT
for cluster PinotCluster, took 318 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 194 ms
TotalProcessed time for event: MessageChange took: 318 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: cffe0e03_DEFAULT. Event session ID: 3000a812cfc0036
Event cffe0e03_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event cffe0e03_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event cffe0e03_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event cffe0e03_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 650921cd_TASK. Event session ID: 3000a812cfc0036
Event 650921cd_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 650921cd_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 650921cd_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 650921cd_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::650921cd_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::650921cd_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 2
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 3ms
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347393, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event cffe0e03_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::cffe0e03_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347423, took 0 ms
Event cffe0e03_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347366 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event cffe0e03_DEFAULT
Event PinotCluster::TASK::650921cd_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::650921cd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347396, took 33 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::650921cd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::650921cd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347430, took 19 ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event cffe0e03_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event cffe0e03_DEFAULT
Event 650921cd_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347368 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event 650921cd_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 650921cd_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 650921cd_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event cffe0e03_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event cffe0e03_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event cffe0e03_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
26 ms for event cffe0e03_DEFAULT
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 32 ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
67 ms for event 650921cd_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 650921cd_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 650921cd_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 650921cd_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 650921cd_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 650921cd_TASK
END: Invoking TASK controller pipeline for event MessageChange::650921cd_TASK for
cluster PinotCluster, took 168 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: e3fb75fd_TASK. Event session ID: 3000a812cfc0036
Event e3fb75fd_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e3fb75fd_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e3fb75fd_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e3fb75fd_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::e3fb75fd_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e3fb75fd_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event cffe0e03_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event cffe0e03_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event cffe0e03_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event cffe0e03_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event cffe0e03_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::e3fb75fd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e3fb75fd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347561, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event cffe0e03_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event cffe0e03_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event cffe0e03_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::cffe0e03_DEFAULT for cluster PinotCluster, took 219 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 460 ms
TotalProcessed time for event: IdealStateChange took: 219 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 898a2f8d_DEFAULT. Event session ID: 3000a812cfc0036
Event 898a2f8d_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 898a2f8d_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 898a2f8d_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 898a2f8d_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::e3fb75fd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e3fb75fd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347585, took 21 ms
Event e3fb75fd_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347536 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event e3fb75fd_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event e3fb75fd_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e3fb75fd_TASK
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347610, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 898a2f8d_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::898a2f8d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347635, took 0 ms
Event 898a2f8d_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347585 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 898a2f8d_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 898a2f8d_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 898a2f8d_DEFAULT
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event e3fb75fd_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e3fb75fd_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e3fb75fd_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e3fb75fd_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e3fb75fd_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e3fb75fd_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::e3fb75fd_TASK
for cluster PinotCluster, took 145 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 589ed36d_TASK. Event session ID: 3000a812cfc0036
Event 589ed36d_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 589ed36d_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 589ed36d_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 589ed36d_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::589ed36d_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::589ed36d_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/4da7d06b-c9e7-46e0-9f2b-
b80c41e58b9a=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/85684e69-338d-4d7f-ba74-
e75db836a216=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 898a2f8d_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 898a2f8d_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 898a2f8d_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Event PinotCluster::TASK::589ed36d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::589ed36d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347709, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 898a2f8d_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::589ed36d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::589ed36d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347738, took 21 ms
Event 589ed36d_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347682 took 78 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 78 ms
for event 589ed36d_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event 898a2f8d_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 589ed36d_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 589ed36d_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 898a2f8d_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 898a2f8d_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 898a2f8d_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 898a2f8d_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 898a2f8d_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 898a2f8d_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 898a2f8d_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::898a2f8d_DEFAULT
for cluster PinotCluster, took 208 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 2 ms
InQueue time for event: MessageChange took: 164 ms
TotalProcessed time for event: MessageChange took: 208 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: e3fb75fd_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event e3fb75fd_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e3fb75fd_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e3fb75fd_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 589ed36d_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 589ed36d_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 589ed36d_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 589ed36d_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 589ed36d_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 589ed36d_TASK
END: Invoking TASK controller pipeline for event MessageChange::589ed36d_TASK for
cluster PinotCluster, took 150 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 08483451_TASK. Event session ID: 3000a812cfc0036
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::08483451_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::08483451_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 08483451_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 08483451_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 08483451_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::08483451_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::08483451_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347833, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e3fb75fd_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e3fb75fd_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110347863, took 0 ms
Event e3fb75fd_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347793 took 75 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 75 ms
for event e3fb75fd_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 24
ms.
Start to refresh stale message cache
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event e3fb75fd_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e3fb75fd_DEFAULT
Event PinotCluster::TASK::08483451_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::08483451_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347877, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::08483451_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::08483451_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110347901, took 21 ms
Event 08483451_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347832 took 91 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 91 ms
for event 08483451_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 08483451_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08483451_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event e3fb75fd_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e3fb75fd_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
2 ms for event e3fb75fd_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e3fb75fd_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Event e3fb75fd_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish computing new ideal states for resources: []
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 08483451_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 08483451_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 08483451_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 08483451_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08483451_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08483451_TASK
END: Invoking TASK controller pipeline for event MessageChange::08483451_TASK for
cluster PinotCluster, took 163 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: fddd475e_TASK. Event session ID: 3000a812cfc0036
Event fddd475e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event fddd475e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event fddd475e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event fddd475e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::fddd475e_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::fddd475e_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 60 ms
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
88 ms for event e3fb75fd_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 44
ms for event e3fb75fd_DEFAULT
Event PinotCluster::TASK::fddd475e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fddd475e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348026, took 64 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event e3fb75fd_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e3fb75fd_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event e3fb75fd_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e3fb75fd_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event e3fb75fd_DEFAULT : Sending Message 64abb2a6-cb13-40d5-99b1-847ff3cc5356 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__25__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event e3fb75fd_DEFAULT : Sending Message 2c94203e-effd-4a75-9e4c-0a9059651408 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__25__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event e3fb75fd_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e3fb75fd_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e3fb75fd_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::e3fb75fd_DEFAULT for cluster PinotCluster, took 312 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 380 ms
TotalProcessed time for event: CurrentStateChange took: 312 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 08483451_DEFAULT. Event session ID: 3000a812cfc0036
Event 08483451_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 08483451_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 08483451_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 08483451_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::08483451_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::08483451_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::fddd475e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fddd475e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348090, took 25 ms
Event fddd475e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110347995 took 121 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 121 ms
for event fddd475e_TASK
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event fddd475e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fddd475e_TASK
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::DEFAULT::08483451_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::08483451_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348131, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 08483451_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::08483451_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::08483451_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348159, took 0 ms
Event 08483451_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348105 took 60 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 60 ms
for event 08483451_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 08483451_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 08483451_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event fddd475e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fddd475e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fddd475e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event fddd475e_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fddd475e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fddd475e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::fddd475e_TASK
for cluster PinotCluster, took 194 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 3c0f5478_TASK. Event session ID: 3000a812cfc0036
Event 3c0f5478_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 3c0f5478_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 3c0f5478_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 3c0f5478_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::3c0f5478_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::3c0f5478_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::3c0f5478_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::3c0f5478_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348214, took 36 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 64 ms for event 08483451_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 08483451_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08483451_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::3c0f5478_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::3c0f5478_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348250, took 20 ms
Event 3c0f5478_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348189 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 82 ms
for event 3c0f5478_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 08483451_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 3c0f5478_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3c0f5478_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 08483451_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 08483451_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08483451_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 08483451_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 08483451_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 08483451_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 08483451_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08483451_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::08483451_DEFAULT
for cluster PinotCluster, took 224 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 401 ms
TotalProcessed time for event: MessageChange took: 224 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: fddd475e_DEFAULT. Event session ID: 3000a812cfc0036
Event fddd475e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event fddd475e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event fddd475e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event fddd475e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 3c0f5478_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 3c0f5478_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 3c0f5478_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 3c0f5478_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3c0f5478_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 3c0f5478_TASK
END: Invoking TASK controller pipeline for event MessageChange::3c0f5478_TASK for
cluster PinotCluster, took 153 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 2346a646_TASK. Event session ID: 3000a812cfc0036
Event 2346a646_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 2346a646_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 2346a646_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 2346a646_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::2346a646_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::2346a646_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348354, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event fddd475e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::fddd475e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348378, took 0 ms
Event fddd475e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348329 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event fddd475e_DEFAULT
Event PinotCluster::TASK::2346a646_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2346a646_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348366, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event fddd475e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event fddd475e_DEFAULT
Event PinotCluster::TASK::2346a646_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2346a646_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110348393, took 20 ms
Event 2346a646_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348342 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event 2346a646_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 2346a646_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2346a646_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event fddd475e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event fddd475e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fddd475e_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event fddd475e_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 2346a646_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2346a646_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2346a646_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event 2346a646_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2346a646_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2346a646_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::2346a646_TASK
for cluster PinotCluster, took 145 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event fddd475e_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event fddd475e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fddd475e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event fddd475e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event fddd475e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event fddd475e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event fddd475e_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fddd475e_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::fddd475e_DEFAULT for cluster PinotCluster, took 207 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 581 ms
TotalProcessed time for event: IdealStateChange took: 207 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 3c0f5478_DEFAULT. Event session ID: 3000a812cfc0036
Event 3c0f5478_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 3c0f5478_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 3c0f5478_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 3c0f5478_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348561, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 3c0f5478_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::3c0f5478_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348585, took 0 ms
Event 3c0f5478_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348536 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 3c0f5478_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 3c0f5478_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 3c0f5478_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 3c0f5478_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 3c0f5478_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3c0f5478_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 3c0f5478_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 3c0f5478_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 3c0f5478_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3c0f5478_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 3c0f5478_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3c0f5478_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 3c0f5478_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 3c0f5478_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 3c0f5478_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::3c0f5478_DEFAULT
for cluster PinotCluster, took 207 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 390 ms
TotalProcessed time for event: MessageChange took: 207 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 2346a646_DEFAULT. Event session ID: 3000a812cfc0036
Event 2346a646_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 2346a646_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 2346a646_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 2346a646_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::2346a646_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::2346a646_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::2346a646_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2346a646_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348768, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 2346a646_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::2346a646_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2346a646_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110348792, took 0 ms
Event 2346a646_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110348743 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 2346a646_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 2346a646_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 2346a646_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 2346a646_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 2346a646_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 2346a646_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2346a646_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 2346a646_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 2346a646_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 59 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 2346a646_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2346a646_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 2346a646_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2346a646_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 2346a646_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2346a646_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2346a646_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::2346a646_DEFAULT for cluster PinotCluster, took 209 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 601 ms
TotalProcessed time for event: CurrentStateChange took: 209 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 167fd6a5_DEFAULT. Event session ID: 3000a812cfc0036
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 167fd6a5_TASK. Event session ID: 3000a812cfc0036
Event 167fd6a5_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 167fd6a5_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 167fd6a5_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 167fd6a5_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 167fd6a5_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 167fd6a5_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 167fd6a5_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 167fd6a5_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::167fd6a5_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::167fd6a5_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/de6cda81-b3d6-4bb1-a879-
17a75a2e2ae9=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/310fcc04-197f-4b5e-a011-
9bf13d7d74eb=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/de6cda81-b3d6-4bb1-a879-
17a75a2e2ae9=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/310fcc04-197f-4b5e-a011-
9bf13d7d74eb=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110349765, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 167fd6a5_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::167fd6a5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110349788, took 0 ms
Event PinotCluster::TASK::167fd6a5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::167fd6a5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110349765, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 167fd6a5_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110349739 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 167fd6a5_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::167fd6a5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::167fd6a5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110349792, took 20 ms
Event 167fd6a5_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110349739 took 74 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 74 ms
for event 167fd6a5_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 167fd6a5_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 167fd6a5_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 167fd6a5_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 167fd6a5_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 167fd6a5_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 167fd6a5_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 167fd6a5_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 167fd6a5_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 167fd6a5_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 167fd6a5_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 167fd6a5_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 167fd6a5_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 167fd6a5_TASK
END: Invoking TASK controller pipeline for event MessageChange::167fd6a5_TASK for
cluster PinotCluster, took 145 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f71aa1e5_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 167fd6a5_DEFAULT
Event PinotCluster::TASK::f71aa1e5_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::f71aa1e5_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event f71aa1e5_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f71aa1e5_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f71aa1e5_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::f71aa1e5_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f71aa1e5_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 167fd6a5_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 167fd6a5_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 167fd6a5_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 167fd6a5_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 167fd6a5_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 167fd6a5_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 167fd6a5_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 167fd6a5_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::167fd6a5_DEFAULT
for cluster PinotCluster, took 205 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 205 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f71aa1e5_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::f71aa1e5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f71aa1e5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110349924, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : Refreshed 11 property IdealState
took 17 ms. Selective: true
Event f71aa1e5_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event f71aa1e5_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event f71aa1e5_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::f71aa1e5_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f71aa1e5_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110349948, took 22 ms
Event f71aa1e5_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110349884 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event f71aa1e5_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event f71aa1e5_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f71aa1e5_TASK
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110349987, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f71aa1e5_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f71aa1e5_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350014, took 0 ms
Event f71aa1e5_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110349944 took 75 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 75 ms
for event f71aa1e5_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event f71aa1e5_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f71aa1e5_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event f71aa1e5_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f71aa1e5_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f71aa1e5_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f71aa1e5_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f71aa1e5_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f71aa1e5_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::f71aa1e5_TASK for
cluster PinotCluster, took 160 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: d24095b2_TASK. Event session ID: 3000a812cfc0036
Event d24095b2_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event d24095b2_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event d24095b2_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event d24095b2_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::d24095b2_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::d24095b2_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event f71aa1e5_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event f71aa1e5_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f71aa1e5_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::d24095b2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d24095b2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350072, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f71aa1e5_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::d24095b2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::d24095b2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350097, took 21 ms
Event d24095b2_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350044 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event d24095b2_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event d24095b2_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event d24095b2_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
94 ms for event f71aa1e5_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event d24095b2_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d24095b2_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event d24095b2_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event d24095b2_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d24095b2_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event d24095b2_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::d24095b2_TASK
for cluster PinotCluster, took 148 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event f71aa1e5_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event f71aa1e5_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f71aa1e5_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event f71aa1e5_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f71aa1e5_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event f71aa1e5_DEFAULT : Sending Message 76c7eee0-e257-4d0d-a5aa-db77f46281ee to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__29__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event f71aa1e5_DEFAULT : Sending Message 058247fa-cd1c-49e9-a189-a4b04406a591 to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__29__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 11 ms for event f71aa1e5_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f71aa1e5_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f71aa1e5_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::f71aa1e5_DEFAULT
for cluster PinotCluster, took 310 ms
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 181 ms
TotalProcessed time for event: MessageChange took: 310 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: d24095b2_DEFAULT. Event session ID: 3000a812cfc0036
Event d24095b2_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event d24095b2_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event d24095b2_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event d24095b2_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 2
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: bfa0c721_TASK. Event session ID: 3000a812cfc0036
Event bfa0c721_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event bfa0c721_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event bfa0c721_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event bfa0c721_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::bfa0c721_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::bfa0c721_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/76c7eee0-e257-4d0d-a5aa-
db77f46281ee=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 2ms
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350282, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event d24095b2_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::d24095b2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350312, took 0 ms
Event d24095b2_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350254 took 65 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 65 ms
for event d24095b2_DEFAULT
Event PinotCluster::TASK::bfa0c721_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::bfa0c721_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350286, took 34 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event d24095b2_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event d24095b2_DEFAULT
Event PinotCluster::TASK::bfa0c721_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::bfa0c721_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350320, took 19 ms
Event bfa0c721_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350258 took 82 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 83 ms
for event bfa0c721_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event bfa0c721_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bfa0c721_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event d24095b2_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event d24095b2_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d24095b2_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event bfa0c721_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bfa0c721_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event bfa0c721_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event bfa0c721_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bfa0c721_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event bfa0c721_TASK
END: Invoking TASK controller pipeline for event MessageChange::bfa0c721_TASK for
cluster PinotCluster, took 154 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 240b2296_TASK. Event session ID: 3000a812cfc0036
Event 240b2296_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 240b2296_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 240b2296_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 240b2296_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::240b2296_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::240b2296_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event d24095b2_DEFAULT
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event d24095b2_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 24
ms for event d24095b2_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d24095b2_DEFAULT
Event PinotCluster::TASK::240b2296_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::240b2296_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350435, took 38 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event d24095b2_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d24095b2_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event d24095b2_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event d24095b2_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event d24095b2_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::d24095b2_DEFAULT for cluster PinotCluster, took 227 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 444 ms
TotalProcessed time for event: IdealStateChange took: 227 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 99317c6a_DEFAULT. Event session ID: 3000a812cfc0036
Event 99317c6a_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 99317c6a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 99317c6a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 99317c6a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::240b2296_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::240b2296_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350473, took 21 ms
Event 240b2296_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350411 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event 240b2296_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 240b2296_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 240b2296_TASK
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350507, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 99317c6a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::99317c6a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350531, took 0 ms
Event 99317c6a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350481 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 99317c6a_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 99317c6a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 99317c6a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 240b2296_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 240b2296_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 240b2296_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 240b2296_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 240b2296_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 240b2296_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::240b2296_TASK
for cluster PinotCluster, took 155 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 99317c6a_TASK. Event session ID: 3000a812cfc0036
Event 99317c6a_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 99317c6a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 99317c6a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 99317c6a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::99317c6a_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::99317c6a_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 99317c6a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 99317c6a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 99317c6a_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::99317c6a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::99317c6a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350591, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 99317c6a_DEFAULT
Event PinotCluster::TASK::99317c6a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::99317c6a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110350616, took 20 ms
Event 99317c6a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350567 took 69 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 69 ms
for event 99317c6a_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 99317c6a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 99317c6a_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 99317c6a_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 99317c6a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 99317c6a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 99317c6a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 99317c6a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 99317c6a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 99317c6a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 99317c6a_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::99317c6a_DEFAULT
for cluster PinotCluster, took 206 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 183 ms
TotalProcessed time for event: MessageChange took: 206 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 240b2296_DEFAULT. Event session ID: 3000a812cfc0036
Event 240b2296_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 240b2296_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 240b2296_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 240b2296_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::240b2296_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::240b2296_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 99317c6a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 99317c6a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 99317c6a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 99317c6a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 99317c6a_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 99317c6a_TASK
END: Invoking TASK controller pipeline for event MessageChange::99317c6a_TASK for
cluster PinotCluster, took 141 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::240b2296_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::240b2296_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350712, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 240b2296_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::240b2296_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::240b2296_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110350737, took 0 ms
Event 240b2296_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110350687 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 240b2296_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 240b2296_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 240b2296_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 240b2296_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 240b2296_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 240b2296_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 240b2296_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 240b2296_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 240b2296_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 56 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 240b2296_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 240b2296_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 240b2296_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 240b2296_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 240b2296_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 240b2296_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 240b2296_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::240b2296_DEFAULT for cluster PinotCluster, took 208 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 394 ms
TotalProcessed time for event: CurrentStateChange took: 208 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 0ca2a6b9_TASK. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 0ca2a6b9_DEFAULT. Event session ID: 3000a812cfc0036
Event 0ca2a6b9_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 0ca2a6b9_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 0ca2a6b9_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 0ca2a6b9_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 0ca2a6b9_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 0ca2a6b9_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 0ca2a6b9_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 0ca2a6b9_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::0ca2a6b9_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::0ca2a6b9_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/16895ec3-2c27-4df9-878c-
bffbc5b7a6cf=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/0544930e-b4ae-4466-9151-
a0103b103ab5=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/60183526-ed1e-49ad-a59e-
abc27d10981f=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/0544930e-b4ae-4466-9151-
a0103b103ab5=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/60183526-ed1e-49ad-a59e-
abc27d10981f=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110351862, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 0ca2a6b9_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::0ca2a6b9_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110351888, took 0 ms
Event PinotCluster::TASK::0ca2a6b9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::0ca2a6b9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110351863, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 0ca2a6b9_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110351835 took 59 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 59 ms
for event 0ca2a6b9_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::0ca2a6b9_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::0ca2a6b9_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110351892, took 20 ms
Event 0ca2a6b9_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110351835 took 78 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 78 ms
for event 0ca2a6b9_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 0ca2a6b9_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 0ca2a6b9_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 0ca2a6b9_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0ca2a6b9_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 0ca2a6b9_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 0ca2a6b9_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 0ca2a6b9_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 0ca2a6b9_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 0ca2a6b9_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 0ca2a6b9_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 0ca2a6b9_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0ca2a6b9_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 0ca2a6b9_TASK
END: Invoking TASK controller pipeline for event MessageChange::0ca2a6b9_TASK for
cluster PinotCluster, took 149 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9f5d6ca2_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 0ca2a6b9_DEFAULT
Event PinotCluster::TASK::9f5d6ca2_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::9f5d6ca2_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 9f5d6ca2_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 9f5d6ca2_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 9f5d6ca2_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::9f5d6ca2_TASK : Refreshed 9 property StateModelDefinition
took 5 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::9f5d6ca2_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event 0ca2a6b9_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 0ca2a6b9_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 0ca2a6b9_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 0ca2a6b9_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 0ca2a6b9_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 0ca2a6b9_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 0ca2a6b9_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 0ca2a6b9_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::0ca2a6b9_DEFAULT
for cluster PinotCluster, took 208 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 208 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9f5d6ca2_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::9f5d6ca2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9f5d6ca2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352028, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : Refreshed 11 property IdealState
took 13 ms. Selective: true
Event 9f5d6ca2_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 9f5d6ca2_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 9f5d6ca2_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::9f5d6ca2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9f5d6ca2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352053, took 21 ms
Event 9f5d6ca2_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110351984 took 91 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 91 ms
for event 9f5d6ca2_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 9f5d6ca2_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9f5d6ca2_TASK
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352083, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 9f5d6ca2_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::9f5d6ca2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352106, took 0 ms
Event 9f5d6ca2_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352043 took 68 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 68 ms
for event 9f5d6ca2_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 9f5d6ca2_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 9f5d6ca2_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 9f5d6ca2_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9f5d6ca2_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9f5d6ca2_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 9f5d6ca2_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9f5d6ca2_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9f5d6ca2_TASK
END: Invoking TASK controller pipeline for event MessageChange::9f5d6ca2_TASK for
cluster PinotCluster, took 162 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 04920c1a_TASK. Event session ID: 3000a812cfc0036
Event 04920c1a_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 04920c1a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 04920c1a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 04920c1a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::04920c1a_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::04920c1a_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 9f5d6ca2_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 9f5d6ca2_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9f5d6ca2_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 9f5d6ca2_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::04920c1a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::04920c1a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352171, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 19 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::04920c1a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::04920c1a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352201, took 20 ms
Event 04920c1a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352146 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event 04920c1a_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 28
ms for event 04920c1a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 04920c1a_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
92 ms for event 9f5d6ca2_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 9f5d6ca2_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event 04920c1a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 04920c1a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 04920c1a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 04920c1a_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 04920c1a_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 04920c1a_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::04920c1a_TASK
for cluster PinotCluster, took 162 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 9f5d6ca2_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9f5d6ca2_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 9f5d6ca2_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 9f5d6ca2_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 9f5d6ca2_DEFAULT : Sending Message 1a84215a-5008-4600-b423-ab4deb93ce33 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__18__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 9f5d6ca2_DEFAULT : Sending Message f0472e45-b36f-467b-ba0a-8fe6c76a756c to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__18__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event 9f5d6ca2_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 9f5d6ca2_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 9f5d6ca2_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::9f5d6ca2_DEFAULT
for cluster PinotCluster, took 291 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 182 ms
TotalProcessed time for event: MessageChange took: 291 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 04920c1a_DEFAULT. Event session ID: 3000a812cfc0036
Event 04920c1a_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 04920c1a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 04920c1a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 04920c1a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9f1be3de_TASK. Event session ID: 3000a812cfc0036
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event 9f1be3de_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 9f1be3de_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 9f1be3de_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 9f1be3de_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::9f1be3de_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::9f1be3de_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/1a84215a-5008-4600-b423-
ab4deb93ce33=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352360, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 04920c1a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::04920c1a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352389, took 0 ms
Event 04920c1a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352334 took 61 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 61 ms
for event 04920c1a_DEFAULT
Event PinotCluster::TASK::9f1be3de_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::9f1be3de_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352363, took 33 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::9f1be3de_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9f1be3de_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352396, took 19 ms
Event 9f1be3de_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352337 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 9f1be3de_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event 04920c1a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 04920c1a_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 9f1be3de_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9f1be3de_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 04920c1a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 04920c1a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 04920c1a_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 9f1be3de_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9f1be3de_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9f1be3de_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 9f1be3de_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9f1be3de_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 9f1be3de_TASK
END: Invoking TASK controller pipeline for event MessageChange::9f1be3de_TASK for
cluster PinotCluster, took 153 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 1a62412e_TASK. Event session ID: 3000a812cfc0036
Event 1a62412e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 1a62412e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 1a62412e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 1a62412e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::1a62412e_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::1a62412e_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
26 ms for event 04920c1a_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 36
ms for event 04920c1a_DEFAULT
Event PinotCluster::TASK::1a62412e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1a62412e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352515, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 04920c1a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 04920c1a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 8 ms
for event 04920c1a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 04920c1a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::1a62412e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1a62412e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352540, took 19 ms
Event 1a62412e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352490 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 1a62412e_TASK
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 04920c1a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 04920c1a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 04920c1a_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::04920c1a_DEFAULT for cluster PinotCluster, took 228 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 428 ms
TotalProcessed time for event: IdealStateChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 051f451d_DEFAULT. Event session ID: 3000a812cfc0036
Event 051f451d_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 051f451d_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 051f451d_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 051f451d_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::051f451d_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::051f451d_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 1a62412e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1a62412e_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::051f451d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::051f451d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352586, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 051f451d_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::051f451d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::051f451d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352610, took 0 ms
Event 051f451d_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352562 took 54 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 54 ms
for event 051f451d_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 1a62412e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1a62412e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1a62412e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 1a62412e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1a62412e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1a62412e_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::1a62412e_TASK
for cluster PinotCluster, took 143 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 051f451d_TASK. Event session ID: 3000a812cfc0036
Event 051f451d_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 051f451d_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 051f451d_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 051f451d_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::051f451d_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 051f451d_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 051f451d_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::051f451d_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::051f451d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::051f451d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352658, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 051f451d_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 051f451d_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 051f451d_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::051f451d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::051f451d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110352682, took 31 ms
Event 051f451d_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352633 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event 051f451d_TASK
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 051f451d_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 051f451d_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 051f451d_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 051f451d_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 051f451d_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 051f451d_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 051f451d_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 051f451d_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 051f451d_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 051f451d_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 051f451d_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::051f451d_DEFAULT
for cluster PinotCluster, took 207 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 174 ms
TotalProcessed time for event: MessageChange took: 207 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 1a62412e_DEFAULT. Event session ID: 3000a812cfc0036
Event 1a62412e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 1a62412e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 1a62412e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 1a62412e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 051f451d_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 051f451d_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 051f451d_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 051f451d_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 051f451d_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 051f451d_TASK
END: Invoking TASK controller pipeline for event MessageChange::051f451d_TASK for
cluster PinotCluster, took 153 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352794, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 1a62412e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1a62412e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110352818, took 0 ms
Event 1a62412e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110352769 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 1a62412e_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 1a62412e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 1a62412e_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 1a62412e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 1a62412e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1a62412e_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1a62412e_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 1a62412e_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 1a62412e_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 60 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 1a62412e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1a62412e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 1a62412e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1a62412e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 1a62412e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1a62412e_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1a62412e_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::1a62412e_DEFAULT for cluster PinotCluster, took 212 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 389 ms
TotalProcessed time for event: CurrentStateChange took: 212 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: de67b433_TASK. Event session ID: 3000a812cfc0036
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: de67b433_DEFAULT. Event session ID: 3000a812cfc0036
Event de67b433_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event de67b433_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event de67b433_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event de67b433_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event de67b433_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event de67b433_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event de67b433_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event de67b433_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::de67b433_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
Event PinotCluster::DEFAULT::de67b433_DEFAULT : Refreshed 9 property
StateModelDefinition took 4 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::de67b433_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::de67b433_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/8ee8fb7c-4029-4010-9f76-
272e7e46659d=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/a9c37dbb-2779-440b-995b-
69db6850aa30=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/8ee8fb7c-4029-4010-9f76-
272e7e46659d=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/a9c37dbb-2779-440b-995b-
69db6850aa30=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::de67b433_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::de67b433_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110353729, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::de67b433_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::de67b433_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110353729, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event de67b433_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::de67b433_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::de67b433_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110353757, took 0 ms
Event de67b433_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110353699 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event de67b433_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::de67b433_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::de67b433_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110353753, took 22 ms
Event de67b433_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110353699 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event de67b433_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event de67b433_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event de67b433_DEFAULT
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event de67b433_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event de67b433_TASK
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event de67b433_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event de67b433_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event de67b433_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event de67b433_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event de67b433_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event de67b433_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event de67b433_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event de67b433_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event de67b433_TASK
END: Invoking TASK controller pipeline for event MessageChange::de67b433_TASK for
cluster PinotCluster, took 152 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1ee519a2_TASK. Event session ID: 3000a812cfc0036
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::1ee519a2_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::1ee519a2_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 1ee519a2_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 1ee519a2_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 1ee519a2_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::1ee519a2_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
30 ms for event de67b433_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::1ee519a2_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event de67b433_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event de67b433_DEFAULT
No throttle config is set!
Event PinotCluster::TASK::1ee519a2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1ee519a2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110353892, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event de67b433_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event de67b433_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event de67b433_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event de67b433_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event de67b433_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event de67b433_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::de67b433_DEFAULT
for cluster PinotCluster, took 228 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1ee519a2_DEFAULT. Event session ID: 3000a812cfc0036
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 1 ms
Event PinotCluster::TASK::1ee519a2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1ee519a2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110353916, took 20 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
Event 1ee519a2_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110353851 took 86 for TASK pipeline
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event 1ee519a2_TASK
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event 1ee519a2_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 1ee519a2_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 1ee519a2_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 1ee519a2_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1ee519a2_TASK
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/1e86230e-f0d5-40cd-bea2-
655f9c5d22cf=-101}
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110353969, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 1ee519a2_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ee519a2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110353993, took 0 ms
Event 1ee519a2_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110353928 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 70 ms
for event 1ee519a2_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 1ee519a2_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1ee519a2_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1ee519a2_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 1ee519a2_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1ee519a2_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1ee519a2_TASK
END: Invoking TASK controller pipeline for event MessageChange::1ee519a2_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: c4c49224_TASK. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 1 ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 1ee519a2_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ee519a2_DEFAULT
Event PinotCluster::TASK::c4c49224_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::c4c49224_TASK : Refreshed 11 property IdealState took 16
ms. Selective: true
Event c4c49224_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c4c49224_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c4c49224_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::c4c49224_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c4c49224_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 1ee519a2_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 1ee519a2_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ee519a2_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::c4c49224_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c4c49224_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354054, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating the new best possible assignment.
Event 1ee519a2_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event 1ee519a2_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Event PinotCluster::TASK::c4c49224_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c4c49224_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354086, took 21 ms
Event c4c49224_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354012 took 96 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 96 ms
for event c4c49224_TASK
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event c4c49224_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c4c49224_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
89 ms for event 1ee519a2_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event c4c49224_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c4c49224_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c4c49224_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event c4c49224_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c4c49224_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c4c49224_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::c4c49224_TASK
for cluster PinotCluster, took 172 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 60cd02d0_TASK. Event session ID: 3000a812cfc0036
Event 60cd02d0_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 60cd02d0_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 60cd02d0_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 60cd02d0_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::60cd02d0_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::60cd02d0_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 1ee519a2_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 1ee519a2_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ee519a2_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 1ee519a2_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ee519a2_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 1ee519a2_DEFAULT : Sending Message 0bc500f1-18e5-4cc8-94bc-33ff1a625545 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__26__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 1ee519a2_DEFAULT : Sending Message 23febb84-8453-4eae-9cbb-787e8caeeabe to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__26__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
Event 1ee519a2_DEFAULT : Sending Message cc9f84a5-b3f1-49b9-8841-821bca9dd926 to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__30__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 1ee519a2_DEFAULT : Sending Message 4b2c910f-a9a8-4a2f-b591-7395b0f5a44e to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__30__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 10 ms for event 1ee519a2_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ee519a2_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ee519a2_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::1ee519a2_DEFAULT
for cluster PinotCluster, took 298 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 106 ms
TotalProcessed time for event: MessageChange took: 298 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: b09ade23_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::60cd02d0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::60cd02d0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354210, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : Refreshed 11 property IdealState
took 17 ms. Selective: true
Event b09ade23_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event b09ade23_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event b09ade23_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::60cd02d0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::60cd02d0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354240, took 22 ms
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
Event 60cd02d0_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354184 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event 60cd02d0_TASK
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 24
ms for event 60cd02d0_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 60cd02d0_TASK
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : 2 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354270, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b09ade23_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b09ade23_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354301, took 0 ms
Event b09ade23_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354226 took 80 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 80 ms
for event b09ade23_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event b09ade23_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b09ade23_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
58 ms for event 60cd02d0_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 60cd02d0_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 60cd02d0_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 60cd02d0_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 60cd02d0_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 60cd02d0_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::60cd02d0_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: b09ade23_TASK. Event session ID: 3000a812cfc0036
Event b09ade23_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event b09ade23_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b09ade23_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b09ade23_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::b09ade23_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b09ade23_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event b09ade23_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event b09ade23_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b09ade23_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event b09ade23_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::b09ade23_TASK : 2 properties refreshed from ZK.
Event PinotCluster::TASK::b09ade23_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354370, took 38 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::b09ade23_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b09ade23_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354408, took 20 ms
Event b09ade23_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354345 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event b09ade23_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 35
ms for event b09ade23_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b09ade23_TASK
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
101 ms for event b09ade23_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event b09ade23_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b09ade23_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b09ade23_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event b09ade23_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b09ade23_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b09ade23_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::b09ade23_TASK
for cluster PinotCluster, took 174 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: aaf65fe6_TASK. Event session ID: 3000a812cfc0036
Event aaf65fe6_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event aaf65fe6_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event aaf65fe6_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event aaf65fe6_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event b09ade23_DEFAULT
Event PinotCluster::TASK::aaf65fe6_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::aaf65fe6_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event b09ade23_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b09ade23_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event b09ade23_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b09ade23_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event b09ade23_DEFAULT : Sending Message e4e9f022-ce92-4738-a59c-ca2b64fdcd25 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__20__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event b09ade23_DEFAULT : Sending Message 9ab87639-b8af-4cda-87f0-f9c3728f94dc to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__20__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event b09ade23_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b09ade23_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b09ade23_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::b09ade23_DEFAULT for cluster PinotCluster, took 320 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 215 ms
TotalProcessed time for event: IdealStateChange took: 320 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: aaf65fe6_DEFAULT. Event session ID: 3000a812cfc0036
Event aaf65fe6_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event aaf65fe6_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event aaf65fe6_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event aaf65fe6_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::aaf65fe6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::aaf65fe6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354547, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 24
ms.
Start to refresh stale message cache
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
Event PinotCluster::TASK::aaf65fe6_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::aaf65fe6_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354573, took 22 ms
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event aaf65fe6_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354519 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event aaf65fe6_TASK
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 5ms
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354577, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event aaf65fe6_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::aaf65fe6_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354607, took 0 ms
Event aaf65fe6_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354547 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event aaf65fe6_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event aaf65fe6_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event aaf65fe6_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event aaf65fe6_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event aaf65fe6_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event aaf65fe6_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event aaf65fe6_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event aaf65fe6_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event aaf65fe6_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event aaf65fe6_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event aaf65fe6_TASK
END: Invoking TASK controller pipeline for event MessageChange::aaf65fe6_TASK for
cluster PinotCluster, took 152 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 756f56fb_TASK. Event session ID: 3000a812cfc0036
Event 756f56fb_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 756f56fb_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 756f56fb_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 756f56fb_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::756f56fb_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::756f56fb_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event aaf65fe6_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event aaf65fe6_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event aaf65fe6_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event aaf65fe6_DEFAULT
Event PinotCluster::TASK::756f56fb_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::756f56fb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354698, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::756f56fb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::756f56fb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354726, took 20 ms
Event 756f56fb_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354671 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event 756f56fb_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event aaf65fe6_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event aaf65fe6_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event aaf65fe6_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 5 ms
for event aaf65fe6_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event aaf65fe6_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 756f56fb_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 756f56fb_TASK
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event aaf65fe6_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event aaf65fe6_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event aaf65fe6_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::aaf65fe6_DEFAULT
for cluster PinotCluster, took 221 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 263 ms
TotalProcessed time for event: MessageChange took: 221 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 756f56fb_DEFAULT. Event session ID: 3000a812cfc0036
Event 756f56fb_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 756f56fb_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 756f56fb_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 756f56fb_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354794, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 756f56fb_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::756f56fb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110354819, took 0 ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 756f56fb_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 756f56fb_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 756f56fb_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 756f56fb_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 756f56fb_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 756f56fb_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::756f56fb_TASK
for cluster PinotCluster, took 149 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 25ffb6ca_TASK. Event session ID: 3000a812cfc0036
Event 25ffb6ca_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 25ffb6ca_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 25ffb6ca_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 25ffb6ca_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::25ffb6ca_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::25ffb6ca_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
Event 756f56fb_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354768 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 756f56fb_DEFAULT
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 756f56fb_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 756f56fb_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::25ffb6ca_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::25ffb6ca_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354846, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::25ffb6ca_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::25ffb6ca_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110354871, took 21 ms
Event 25ffb6ca_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354820 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event 25ffb6ca_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 756f56fb_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 756f56fb_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 756f56fb_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 756f56fb_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 25ffb6ca_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 25ffb6ca_TASK
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
30 ms for event 756f56fb_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 61 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 756f56fb_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 25ffb6ca_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 25ffb6ca_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 25ffb6ca_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 25ffb6ca_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 25ffb6ca_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 25ffb6ca_TASK
END: Invoking TASK controller pipeline for event MessageChange::25ffb6ca_TASK for
cluster PinotCluster, took 149 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 756f56fb_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 756f56fb_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 756f56fb_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 756f56fb_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 756f56fb_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 756f56fb_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 756f56fb_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::756f56fb_DEFAULT for cluster PinotCluster, took 222 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 1 ms
InQueue time for event: CurrentStateChange took: 173 ms
TotalProcessed time for event: CurrentStateChange took: 222 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 25ffb6ca_DEFAULT. Event session ID: 3000a812cfc0036
Event 25ffb6ca_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 25ffb6ca_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 25ffb6ca_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 25ffb6ca_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355017, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 25ffb6ca_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::25ffb6ca_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355042, took 1 ms
Event 25ffb6ca_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110354990 took 59 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 59 ms
for event 25ffb6ca_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 25ffb6ca_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 25ffb6ca_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 25ffb6ca_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 25ffb6ca_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 25ffb6ca_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 25ffb6ca_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 25ffb6ca_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 25ffb6ca_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 25ffb6ca_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 25ffb6ca_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 25ffb6ca_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 25ffb6ca_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 25ffb6ca_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 25ffb6ca_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::25ffb6ca_DEFAULT
for cluster PinotCluster, took 218 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 391 ms
TotalProcessed time for event: MessageChange took: 218 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: af9b1262_TASK. Event session ID: 3000a812cfc0036
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: af9b1262_DEFAULT. Event session ID: 3000a812cfc0036
Event af9b1262_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event af9b1262_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event af9b1262_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event af9b1262_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event af9b1262_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event af9b1262_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event af9b1262_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event af9b1262_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::af9b1262_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::af9b1262_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/030a25db-bdc8-4870-a247-
ce3da83911c6=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/be85b721-ca91-4a96-8bfd-
fed9f7d91faf=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/6dfdc312-4f56-4532-bebf-
7f0cf1e038a8=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/030a25db-bdc8-4870-a247-
ce3da83911c6=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/be85b721-ca91-4a96-8bfd-
fed9f7d91faf=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/6dfdc312-4f56-4532-bebf-
7f0cf1e038a8=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::af9b1262_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::af9b1262_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355668, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355669, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event af9b1262_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::af9b1262_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355698, took 0 ms
Event af9b1262_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110355638 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event af9b1262_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::af9b1262_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::af9b1262_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355694, took 24 ms
Event af9b1262_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110355638 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event af9b1262_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event af9b1262_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event af9b1262_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event af9b1262_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event af9b1262_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event af9b1262_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event af9b1262_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event af9b1262_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event af9b1262_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event af9b1262_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event af9b1262_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event af9b1262_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event af9b1262_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event af9b1262_TASK
END: Invoking TASK controller pipeline for event MessageChange::af9b1262_TASK for
cluster PinotCluster, took 153 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
Start emergency rebalance.
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ac5d019a_TASK. Event session ID: 3000a812cfc0036
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event af9b1262_DEFAULT
Event PinotCluster::TASK::ac5d019a_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::ac5d019a_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event ac5d019a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ac5d019a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ac5d019a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::ac5d019a_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ac5d019a_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event af9b1262_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event af9b1262_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event af9b1262_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event af9b1262_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event af9b1262_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::ac5d019a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ac5d019a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355831, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event af9b1262_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event af9b1262_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event af9b1262_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::af9b1262_DEFAULT
for cluster PinotCluster, took 219 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 0 ms
TotalProcessed time for event: MessageChange took: 219 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ac5d019a_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event ac5d019a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ac5d019a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ac5d019a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::ac5d019a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ac5d019a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355855, took 22 ms
START: InstanceMessagesCache.refresh()
Event ac5d019a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110355791 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event ac5d019a_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event ac5d019a_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ac5d019a_TASK
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355897, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ac5d019a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ac5d019a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110355921, took 0 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event ac5d019a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110355857 took 69 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 69 ms
for event ac5d019a_DEFAULT
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event ac5d019a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ac5d019a_DEFAULT
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event ac5d019a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ac5d019a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ac5d019a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ac5d019a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ac5d019a_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ac5d019a_TASK
END: Invoking TASK controller pipeline for event MessageChange::ac5d019a_TASK for
cluster PinotCluster, took 158 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: a9090533_TASK. Event session ID: 3000a812cfc0036
Event a9090533_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event a9090533_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event a9090533_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event a9090533_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::a9090533_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::a9090533_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Event PinotCluster::TASK::a9090533_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a9090533_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355974, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 4ms
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event ac5d019a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ac5d019a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ac5d019a_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event ac5d019a_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
Event PinotCluster::TASK::a9090533_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a9090533_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110355999, took 32 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Event a9090533_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110355949 took 83 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 83 ms
for event a9090533_TASK
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event a9090533_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a9090533_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
88 ms for event ac5d019a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event a9090533_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event a9090533_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a9090533_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event a9090533_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a9090533_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a9090533_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event IdealStateChange::a9090533_TASK
for cluster PinotCluster, took 158 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 561a9368_TASK. Event session ID: 3000a812cfc0036
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event ac5d019a_DEFAULT
Event PinotCluster::TASK::561a9368_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::561a9368_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 561a9368_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 561a9368_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 561a9368_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::561a9368_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::561a9368_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event ac5d019a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ac5d019a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event ac5d019a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ac5d019a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 1 ms
Event ac5d019a_DEFAULT : Sending Message da1de03e-4b9f-4eba-a2ce-453eb2228b7d to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__17__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event ac5d019a_DEFAULT : Sending Message f28d0fd1-200c-4c0d-8847-edab95b975a0 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__17__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 10 ms for event ac5d019a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ac5d019a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ac5d019a_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::ac5d019a_DEFAULT
for cluster PinotCluster, took 291 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 192 ms
TotalProcessed time for event: MessageChange took: 291 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 8590838a_DEFAULT. Event session ID: 3000a812cfc0036
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::8590838a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8590838a_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event 8590838a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 8590838a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 8590838a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::8590838a_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::8590838a_DEFAULT : Refreshed 1 property
ClusterConstraint took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
Event PinotCluster::TASK::561a9368_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::561a9368_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356151, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 24
ms.
Start to refresh stale message cache
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::TASK::561a9368_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::561a9368_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356181, took 20 ms
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event 561a9368_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356107 took 95 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 95 ms
for event 561a9368_TASK
Event PinotCluster::DEFAULT::8590838a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8590838a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356192, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 8590838a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::8590838a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8590838a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356220, took 0 ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 561a9368_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 561a9368_TASK
Event 8590838a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356148 took 77 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 77 ms
for event 8590838a_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 8590838a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8590838a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 561a9368_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 561a9368_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 561a9368_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 561a9368_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 561a9368_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 561a9368_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::561a9368_TASK for
cluster PinotCluster, took 168 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 8590838a_TASK. Event session ID: 3000a812cfc0036
Event 8590838a_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 8590838a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8590838a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8590838a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::8590838a_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8590838a_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 8590838a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 8590838a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8590838a_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event 8590838a_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Computing BestPossibleMapping for amxordersntt_REALTIME
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::8590838a_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::8590838a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356300, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
Event PinotCluster::TASK::8590838a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8590838a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356327, took 20 ms
Event 8590838a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356275 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event 8590838a_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 33
ms for event 8590838a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8590838a_TASK
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
97 ms for event 8590838a_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 8590838a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 8590838a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 8590838a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8590838a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8590838a_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8590838a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8590838a_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::8590838a_TASK
for cluster PinotCluster, took 160 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 08b2f7ab_TASK. Event session ID: 3000a812cfc0036
Event 08b2f7ab_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 08b2f7ab_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 08b2f7ab_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 08b2f7ab_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 8590838a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8590838a_DEFAULT
Event PinotCluster::TASK::08b2f7ab_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::08b2f7ab_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 8590838a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8590838a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 8590838a_DEFAULT : Sending Message 40ee4520-136d-49b7-ad85-48c2dbb6a157 to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__8__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 8590838a_DEFAULT : Sending Message 74df0fda-3b21-4bcd-baec-ef3e505e9654 to
Server_pinot-server-11.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__8__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 8590838a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8590838a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8590838a_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::8590838a_DEFAULT for cluster PinotCluster, took 305 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 153 ms
TotalProcessed time for event: IdealStateChange took: 305 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 08b2f7ab_DEFAULT. Event session ID: 3000a812cfc0036
Event 08b2f7ab_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 08b2f7ab_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 08b2f7ab_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 08b2f7ab_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 42, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 42, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0044
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
Event PinotCluster::TASK::08b2f7ab_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::08b2f7ab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356464, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
103 START: CallbackHandler 55, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 55 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler55, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 55, /PinotCluster/INSTANCES/Server_pinot-server-
11.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356478, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 08b2f7ab_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::08b2f7ab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356508, took 0 ms
Event 08b2f7ab_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356453 took 62 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event 08b2f7ab_DEFAULT
Event PinotCluster::TASK::08b2f7ab_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::08b2f7ab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356491, took 29 ms
Event 08b2f7ab_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356435 took 86 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event 08b2f7ab_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 08b2f7ab_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 08b2f7ab_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 08b2f7ab_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08b2f7ab_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 08b2f7ab_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 08b2f7ab_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 08b2f7ab_TASK
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08b2f7ab_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 08b2f7ab_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 08b2f7ab_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 08b2f7ab_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08b2f7ab_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 08b2f7ab_TASK
END: Invoking TASK controller pipeline for event MessageChange::08b2f7ab_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8f2009f7_TASK. Event session ID: 3000a812cfc0036
Event 8f2009f7_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 8f2009f7_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8f2009f7_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8f2009f7_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::8f2009f7_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8f2009f7_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
29 ms for event 08b2f7ab_DEFAULT
Event PinotCluster::TASK::8f2009f7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8f2009f7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356617, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 08b2f7ab_DEFAULT
Event PinotCluster::TASK::8f2009f7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8f2009f7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356641, took 20 ms
Event 8f2009f7_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356592 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 8f2009f7_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 08b2f7ab_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08b2f7ab_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 08b2f7ab_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 08b2f7ab_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 08b2f7ab_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 08b2f7ab_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 08b2f7ab_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::08b2f7ab_DEFAULT
for cluster PinotCluster, took 222 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 253 ms
TotalProcessed time for event: MessageChange took: 222 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8f2009f7_DEFAULT. Event session ID: 3000a812cfc0036
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Event 8f2009f7_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 8f2009f7_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 8f2009f7_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 8f2009f7_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 8f2009f7_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8f2009f7_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356700, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 8f2009f7_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8f2009f7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356724, took 0 ms
Event 8f2009f7_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356675 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 8f2009f7_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
57 ms for event 8f2009f7_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8f2009f7_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8f2009f7_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8f2009f7_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8f2009f7_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8f2009f7_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::8f2009f7_TASK
for cluster PinotCluster, took 146 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 39bda919_TASK. Event session ID: 3000a812cfc0036
Event 39bda919_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 39bda919_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 39bda919_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 39bda919_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::39bda919_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::39bda919_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 8f2009f7_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 8f2009f7_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::39bda919_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::39bda919_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356763, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 8f2009f7_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 8f2009f7_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8f2009f7_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 4
ms for event 8f2009f7_DEFAULT
Start computing new ideal states for resources: []
Event PinotCluster::TASK::39bda919_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::39bda919_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110356788, took 20 ms
Event 39bda919_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356738 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 39bda919_TASK
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 17
ms for event 39bda919_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 39bda919_TASK
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 8f2009f7_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 8f2009f7_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 58 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 8f2009f7_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8f2009f7_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 8f2009f7_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8f2009f7_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 39bda919_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 39bda919_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 39bda919_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 39bda919_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 39bda919_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 39bda919_TASK
END: Invoking TASK controller pipeline for event MessageChange::39bda919_TASK for
cluster PinotCluster, took 141 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 8f2009f7_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8f2009f7_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8f2009f7_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::8f2009f7_DEFAULT for cluster PinotCluster, took 208 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 186 ms
TotalProcessed time for event: CurrentStateChange took: 208 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 39bda919_DEFAULT. Event session ID: 3000a812cfc0036
Event 39bda919_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 39bda919_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 39bda919_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 39bda919_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::39bda919_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::39bda919_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::39bda919_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::39bda919_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356909, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 39bda919_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::39bda919_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::39bda919_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110356933, took 0 ms
Event 39bda919_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110356883 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 39bda919_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 39bda919_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 39bda919_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 39bda919_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 39bda919_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 39bda919_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 39bda919_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 39bda919_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 39bda919_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 39bda919_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 39bda919_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 39bda919_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 39bda919_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 39bda919_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 39bda919_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::39bda919_DEFAULT
for cluster PinotCluster, took 208 ms
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 390 ms
TotalProcessed time for event: MessageChange took: 208 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: c29413b0_TASK. Event session ID: 3000a812cfc0036
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: c29413b0_DEFAULT. Event session ID: 3000a812cfc0036
Event c29413b0_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event c29413b0_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c29413b0_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c29413b0_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event c29413b0_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event c29413b0_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event c29413b0_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event c29413b0_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::c29413b0_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c29413b0_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/a3487c78-9c3e-4668-89e7-
1862cc17b77b=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/76db93f8-aec8-4f45-ae93-
a24bec95ec08=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/76db93f8-aec8-4f45-ae93-
a24bec95ec08=-101}
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357347, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c29413b0_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::c29413b0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357376, took 0 ms
Event PinotCluster::TASK::c29413b0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c29413b0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357348, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event c29413b0_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357316 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event c29413b0_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::c29413b0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c29413b0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357379, took 20 ms
Event c29413b0_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357316 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event c29413b0_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event c29413b0_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event c29413b0_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event c29413b0_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c29413b0_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event c29413b0_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event c29413b0_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event c29413b0_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event c29413b0_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event c29413b0_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c29413b0_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event c29413b0_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c29413b0_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c29413b0_TASK
END: Invoking TASK controller pipeline for event MessageChange::c29413b0_TASK for
cluster PinotCluster, took 155 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b23a28af_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event c29413b0_DEFAULT
Event PinotCluster::TASK::b23a28af_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::b23a28af_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event b23a28af_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b23a28af_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b23a28af_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::b23a28af_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b23a28af_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event c29413b0_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event c29413b0_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event c29413b0_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event c29413b0_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c29413b0_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event c29413b0_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event c29413b0_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event c29413b0_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::c29413b0_DEFAULT
for cluster PinotCluster, took 218 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 218 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b23a28af_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::b23a28af_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b23a28af_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357511, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event b23a28af_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event b23a28af_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event b23a28af_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::b23a28af_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b23a28af_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357535, took 23 ms
Event b23a28af_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357471 took 88 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 88 ms
for event b23a28af_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 21
ms for event b23a28af_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b23a28af_TASK
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357576, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b23a28af_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b23a28af_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357601, took 0 ms
Event b23a28af_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357534 took 72 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 72 ms
for event b23a28af_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event b23a28af_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event b23a28af_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event b23a28af_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b23a28af_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b23a28af_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event b23a28af_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b23a28af_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b23a28af_TASK
END: Invoking TASK controller pipeline for event MessageChange::b23a28af_TASK for
cluster PinotCluster, took 164 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 77b8636c_TASK. Event session ID: 3000a812cfc0036
Event 77b8636c_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 77b8636c_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 77b8636c_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 77b8636c_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::77b8636c_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::77b8636c_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event b23a28af_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event b23a28af_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b23a28af_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event PinotCluster::TASK::77b8636c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::77b8636c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357661, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b23a28af_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Computing BestPossibleMapping for amxordersntt_REALTIME
Global baseline calculation completed and has been persisted into metadata store.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::77b8636c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::77b8636c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357685, took 20 ms
Event 77b8636c_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357636 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 77b8636c_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 77b8636c_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 77b8636c_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
82 ms for event b23a28af_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 77b8636c_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 77b8636c_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 77b8636c_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 77b8636c_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 77b8636c_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 77b8636c_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::77b8636c_TASK
for cluster PinotCluster, took 141 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event b23a28af_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event b23a28af_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b23a28af_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event b23a28af_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b23a28af_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event b23a28af_DEFAULT : Sending Message 37d3414c-5451-4558-a78b-bca07db44b96 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__10__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event b23a28af_DEFAULT : Sending Message eedbeb8b-3c64-4e8c-a029-6cd0995f0787 to
Server_pinot-server-13.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__10__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event b23a28af_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b23a28af_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b23a28af_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::b23a28af_DEFAULT
for cluster PinotCluster, took 285 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 192 ms
TotalProcessed time for event: MessageChange took: 285 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 77b8636c_DEFAULT. Event session ID: 3000a812cfc0036
Event 77b8636c_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 77b8636c_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 77b8636c_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 77b8636c_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 8bf92481_TASK. Event session ID: 3000a812cfc0036
Event 8bf92481_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 8bf92481_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8bf92481_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8bf92481_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::8bf92481_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8bf92481_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/37d3414c-5451-4558-a78b-
bca07db44b96=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 24
ms.
Start to refresh stale message cache
103 START: CallbackHandler 24, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 24, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb004c
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 68, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 68 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler68, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 5
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 68, /PinotCluster/INSTANCES/Server_pinot-server-
13.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 6ms
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357849, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 77b8636c_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::77b8636c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110357879, took 0 ms
Event 77b8636c_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357819 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event 77b8636c_DEFAULT
Event PinotCluster::TASK::8bf92481_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::8bf92481_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357853, took 33 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 77b8636c_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 77b8636c_DEFAULT
Event PinotCluster::TASK::8bf92481_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8bf92481_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110357886, took 20 ms
Event 8bf92481_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357822 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 85 ms
for event 8bf92481_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 8bf92481_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8bf92481_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 77b8636c_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 77b8636c_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 77b8636c_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 8bf92481_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8bf92481_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8bf92481_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8bf92481_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8bf92481_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8bf92481_TASK
END: Invoking TASK controller pipeline for event MessageChange::8bf92481_TASK for
cluster PinotCluster, took 155 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: f3090533_TASK. Event session ID: 3000a812cfc0036
Event f3090533_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f3090533_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f3090533_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f3090533_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::f3090533_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f3090533_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 77b8636c_DEFAULT
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 77b8636c_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 77b8636c_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 77b8636c_DEFAULT
Event PinotCluster::TASK::f3090533_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f3090533_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358004, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 77b8636c_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 77b8636c_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 77b8636c_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 77b8636c_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 77b8636c_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::77b8636c_DEFAULT for cluster PinotCluster, took 218 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 429 ms
TotalProcessed time for event: IdealStateChange took: 218 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 2ea989db_DEFAULT. Event session ID: 3000a812cfc0036
Event 2ea989db_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 2ea989db_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 2ea989db_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 2ea989db_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::f3090533_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f3090533_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358029, took 21 ms
Event f3090533_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110357978 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event f3090533_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event f3090533_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f3090533_TASK
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358064, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 2ea989db_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2ea989db_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358089, took 0 ms
Event 2ea989db_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358038 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 2ea989db_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 2ea989db_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2ea989db_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event f3090533_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f3090533_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f3090533_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f3090533_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f3090533_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f3090533_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::f3090533_TASK
for cluster PinotCluster, took 146 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 2ea989db_TASK. Event session ID: 3000a812cfc0036
Event 2ea989db_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 2ea989db_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 2ea989db_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 2ea989db_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::2ea989db_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::2ea989db_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 32
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 71 ms for event 2ea989db_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 2ea989db_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2ea989db_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::2ea989db_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2ea989db_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358162, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::2ea989db_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2ea989db_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358187, took 21 ms
Event 2ea989db_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358124 took 85 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 85 ms
for event 2ea989db_TASK
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 2ea989db_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 2ea989db_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2ea989db_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event 2ea989db_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 2ea989db_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2ea989db_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 2ea989db_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2ea989db_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 2ea989db_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2ea989db_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2ea989db_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::2ea989db_DEFAULT
for cluster PinotCluster, took 228 ms
Callback time for event: MessageChange took: 5 ms
InQueue time for event: MessageChange took: 162 ms
TotalProcessed time for event: MessageChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: f3090533_DEFAULT. Event session ID: 3000a812cfc0036
Event f3090533_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event f3090533_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event f3090533_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event f3090533_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::f3090533_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::f3090533_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 2ea989db_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2ea989db_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2ea989db_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 2ea989db_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2ea989db_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2ea989db_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::2ea989db_TASK for
cluster PinotCluster, took 157 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::f3090533_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f3090533_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358292, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f3090533_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::f3090533_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f3090533_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358317, took 0 ms
Event f3090533_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358267 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event f3090533_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event f3090533_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f3090533_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event f3090533_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event f3090533_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f3090533_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f3090533_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event f3090533_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event f3090533_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 56 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event f3090533_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f3090533_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event f3090533_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f3090533_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event f3090533_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f3090533_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f3090533_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::f3090533_DEFAULT for cluster PinotCluster, took 208 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 399 ms
TotalProcessed time for event: CurrentStateChange took: 208 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 21409296_DEFAULT. Event session ID: 3000a812cfc0036
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 21409296_TASK. Event session ID: 3000a812cfc0036
Event 21409296_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 21409296_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 21409296_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 21409296_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 21409296_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 21409296_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 21409296_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 21409296_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::21409296_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event PinotCluster::TASK::21409296_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::21409296_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::21409296_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
START: InstanceMessagesCache.refresh()
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/3230f2f9-6fd8-4a81-9d34-
273c6ae675c5=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/64f0d83e-dfc4-4ae8-ba4f-
f8aa99b624e2=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/820ee016-9088-42fa-ba38-
9a7b3f250cbe=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/3230f2f9-6fd8-4a81-9d34-
273c6ae675c5=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/64f0d83e-dfc4-4ae8-ba4f-
f8aa99b624e2=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/820ee016-9088-42fa-ba38-
9a7b3f250cbe=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::21409296_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::21409296_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358710, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 21409296_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::21409296_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::21409296_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358734, took 0 ms
Event PinotCluster::TASK::21409296_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::21409296_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358710, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 21409296_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358683 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event 21409296_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::21409296_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::21409296_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358738, took 20 ms
Event 21409296_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358683 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event 21409296_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 21409296_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 21409296_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 21409296_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 21409296_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 21409296_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 21409296_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 21409296_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
73 ms for event 21409296_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 21409296_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 21409296_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 21409296_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 21409296_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 21409296_TASK
END: Invoking TASK controller pipeline for event MessageChange::21409296_TASK for
cluster PinotCluster, took 167 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ab1a2739_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 39 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
42 ms for event 21409296_DEFAULT
Event PinotCluster::TASK::ab1a2739_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::ab1a2739_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event ab1a2739_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ab1a2739_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ab1a2739_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::ab1a2739_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ab1a2739_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 21409296_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 21409296_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 21409296_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 21409296_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 21409296_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 21409296_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 21409296_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 21409296_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::21409296_DEFAULT
for cluster PinotCluster, took 226 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 226 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ab1a2739_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::ab1a2739_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ab1a2739_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358890, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : Refreshed 11 property IdealState
took 17 ms. Selective: true
Event ab1a2739_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ab1a2739_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ab1a2739_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::ab1a2739_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ab1a2739_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110358915, took 21 ms
Event ab1a2739_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358850 took 87 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 87 ms
for event ab1a2739_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event ab1a2739_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event ab1a2739_TASK
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358952, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ab1a2739_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ab1a2739_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110358976, took 0 ms
Event ab1a2739_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110358909 took 72 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 72 ms
for event ab1a2739_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event ab1a2739_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ab1a2739_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event ab1a2739_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event ab1a2739_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ab1a2739_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ab1a2739_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ab1a2739_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ab1a2739_TASK
END: Invoking TASK controller pipeline for event MessageChange::ab1a2739_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 7b30df67_TASK. Event session ID: 3000a812cfc0036
Event 7b30df67_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 7b30df67_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 7b30df67_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 7b30df67_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::7b30df67_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::7b30df67_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::7b30df67_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7b30df67_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359037, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 62 ms for event ab1a2739_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ab1a2739_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ab1a2739_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Event ab1a2739_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating the new best possible assignment.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Global baseline calculation completed and has been persisted into metadata store.
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::7b30df67_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7b30df67_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359062, took 20 ms
Event 7b30df67_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359011 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event 7b30df67_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 7b30df67_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7b30df67_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
96 ms for event ab1a2739_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
68 ms for event 7b30df67_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7b30df67_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7b30df67_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 7b30df67_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7b30df67_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7b30df67_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::7b30df67_TASK
for cluster PinotCluster, took 159 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event ab1a2739_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event ab1a2739_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ab1a2739_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event ab1a2739_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ab1a2739_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event ab1a2739_DEFAULT : Sending Message ecbecb93-27e5-4122-b0c9-4cc21ec21177 to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__27__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event ab1a2739_DEFAULT : Sending Message 5f1c41ea-d4d6-4a54-aa8e-237a23a2386d to
Server_pinot-server-9.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__27__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event ab1a2739_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ab1a2739_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ab1a2739_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::ab1a2739_DEFAULT
for cluster PinotCluster, took 309 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 201 ms
TotalProcessed time for event: MessageChange took: 309 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 7b30df67_DEFAULT. Event session ID: 3000a812cfc0036
Event 7b30df67_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 7b30df67_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 7b30df67_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 7b30df67_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 9414d07e_TASK. Event session ID: 3000a812cfc0036
Event 9414d07e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 9414d07e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 9414d07e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 9414d07e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::9414d07e_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::9414d07e_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/ecbecb93-27e5-4122-b0c9-
4cc21ec21177=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
103 START: CallbackHandler 50, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 50, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/2000a812526002c listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 58, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 58 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler58, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 58, /PinotCluster/INSTANCES/Server_pinot-server-
9.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359246, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7b30df67_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7b30df67_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359274, took 0 ms
Event PinotCluster::TASK::9414d07e_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::9414d07e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359249, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7b30df67_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359218 took 63 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 63 ms
for event 7b30df67_DEFAULT
Event PinotCluster::TASK::9414d07e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::9414d07e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359281, took 20 ms
Event 9414d07e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359221 took 80 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 80 ms
for event 9414d07e_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 7b30df67_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7b30df67_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 9414d07e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 9414d07e_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 7b30df67_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 7b30df67_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7b30df67_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 7b30df67_DEFAULT
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 25 ms
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
73 ms for event 9414d07e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9414d07e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 9414d07e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 1
ms for event 9414d07e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9414d07e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 9414d07e_TASK
END: Invoking TASK controller pipeline for event MessageChange::9414d07e_TASK for
cluster PinotCluster, took 175 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: f1ac1c94_TASK. Event session ID: 3000a812cfc0036
Event f1ac1c94_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f1ac1c94_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f1ac1c94_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f1ac1c94_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::f1ac1c94_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f1ac1c94_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 7b30df67_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 7b30df67_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7b30df67_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 7b30df67_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7b30df67_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 7b30df67_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7b30df67_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7b30df67_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::7b30df67_DEFAULT for cluster PinotCluster, took 221 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 462 ms
TotalProcessed time for event: IdealStateChange took: 221 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Controller stopping periodical rebalance timer at period 9223372036854775807
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ef845351_DEFAULT. Event session ID: 3000a812cfc0036
Event ef845351_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event ef845351_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event ef845351_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event ef845351_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::ef845351_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::f1ac1c94_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f1ac1c94_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359421, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::ef845351_DEFAULT : Refreshed 1 property
ClusterConstraint took 4 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::f1ac1c94_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f1ac1c94_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359445, took 22 ms
Event f1ac1c94_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359396 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 71 ms
for event f1ac1c94_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event f1ac1c94_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event f1ac1c94_TASK
Event PinotCluster::DEFAULT::ef845351_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ef845351_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359469, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event ef845351_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::ef845351_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::ef845351_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359493, took 0 ms
Event ef845351_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359439 took 60 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 60 ms
for event ef845351_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event ef845351_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event ef845351_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event f1ac1c94_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event f1ac1c94_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f1ac1c94_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f1ac1c94_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f1ac1c94_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f1ac1c94_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event CurrentStateChange::f1ac1c94_TASK
for cluster PinotCluster, took 147 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: ef845351_TASK. Event session ID: 3000a812cfc0036
Event ef845351_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event ef845351_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event ef845351_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event ef845351_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::ef845351_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::ef845351_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 59 ms for event ef845351_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event ef845351_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ef845351_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::ef845351_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ef845351_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359569, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event ef845351_DEFAULT
Event PinotCluster::TASK::ef845351_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::ef845351_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110359597, took 20 ms
Event ef845351_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359543 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event ef845351_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event ef845351_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event ef845351_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ef845351_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event ef845351_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event ef845351_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event ef845351_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ef845351_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event ef845351_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event ef845351_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event ef845351_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::ef845351_DEFAULT
for cluster PinotCluster, took 222 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 167 ms
TotalProcessed time for event: MessageChange took: 222 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: f1ac1c94_DEFAULT. Event session ID: 3000a812cfc0036
Event f1ac1c94_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event f1ac1c94_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event f1ac1c94_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event f1ac1c94_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event ef845351_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ef845351_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event ef845351_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event ef845351_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ef845351_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event ef845351_TASK
END: Invoking TASK controller pipeline for event MessageChange::ef845351_TASK for
cluster PinotCluster, took 150 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359687, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event f1ac1c94_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::f1ac1c94_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110359712, took 0 ms
Event f1ac1c94_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110359661 took 58 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 58 ms
for event f1ac1c94_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event f1ac1c94_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f1ac1c94_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event f1ac1c94_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event f1ac1c94_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f1ac1c94_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f1ac1c94_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event f1ac1c94_DEFAULT
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 26 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event f1ac1c94_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 60 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event f1ac1c94_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f1ac1c94_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event f1ac1c94_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f1ac1c94_DEFAULT
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event f1ac1c94_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event f1ac1c94_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event f1ac1c94_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::f1ac1c94_DEFAULT for cluster PinotCluster, took 215 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 397 ms
TotalProcessed time for event: CurrentStateChange took: 215 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 500bb7eb_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
Event 500bb7eb_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 500bb7eb_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 500bb7eb_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 500bb7eb_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 500bb7eb_TASK. Event session ID: 3000a812cfc0036
Event 500bb7eb_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 500bb7eb_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 500bb7eb_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 500bb7eb_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::500bb7eb_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::500bb7eb_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e9ca3e19-c54a-477f-a5b8-
360c23366e66=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/bd31c3bf-95be-4fdd-ad1b-
a85c214410ad=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/ef8fb0b6-db30-42ad-9037-
d9b0acb0cbfd=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e9ca3e19-c54a-477f-a5b8-
360c23366e66=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/bd31c3bf-95be-4fdd-ad1b-
a85c214410ad=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/ef8fb0b6-db30-42ad-9037-
d9b0acb0cbfd=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362122, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 500bb7eb_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::500bb7eb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362148, took 0 ms
Event PinotCluster::TASK::500bb7eb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::500bb7eb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362123, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 500bb7eb_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362094 took 61 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 61 ms
for event 500bb7eb_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::500bb7eb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::500bb7eb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362153, took 30 ms
Event 500bb7eb_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362094 took 90 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 90 ms
for event 500bb7eb_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
30 ms for event 500bb7eb_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 500bb7eb_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 500bb7eb_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 500bb7eb_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 500bb7eb_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 500bb7eb_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
5 ms for event 500bb7eb_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 500bb7eb_TASK
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 500bb7eb_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 500bb7eb_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 500bb7eb_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 500bb7eb_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 500bb7eb_TASK
END: Invoking TASK controller pipeline for event MessageChange::500bb7eb_TASK for
cluster PinotCluster, took 166 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 17a5d043_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 500bb7eb_DEFAULT
Event PinotCluster::TASK::17a5d043_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::17a5d043_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 17a5d043_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 17a5d043_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 17a5d043_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::17a5d043_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::17a5d043_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 500bb7eb_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 500bb7eb_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 500bb7eb_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 500bb7eb_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 500bb7eb_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 500bb7eb_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 500bb7eb_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 500bb7eb_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::500bb7eb_DEFAULT
for cluster PinotCluster, took 233 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 233 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 17a5d043_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::17a5d043_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::17a5d043_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362302, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event 17a5d043_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 17a5d043_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 17a5d043_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::17a5d043_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::17a5d043_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362330, took 24 ms
Event 17a5d043_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362260 took 95 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 95 ms
for event 17a5d043_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 17a5d043_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 17a5d043_TASK
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362367, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 17a5d043_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::17a5d043_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362392, took 0 ms
Event 17a5d043_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362327 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 70 ms
for event 17a5d043_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 17a5d043_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 17a5d043_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 17a5d043_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 17a5d043_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 17a5d043_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 17a5d043_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 17a5d043_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 17a5d043_TASK
END: Invoking TASK controller pipeline for event MessageChange::17a5d043_TASK for
cluster PinotCluster, took 168 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 67874928_TASK. Event session ID: 3000a812cfc0036
Event 67874928_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 67874928_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 67874928_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 67874928_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::67874928_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::67874928_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 34
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 69 ms for event 17a5d043_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 17a5d043_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 17a5d043_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::67874928_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::67874928_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362467, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 17a5d043_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 19 ms
Event PinotCluster::TASK::67874928_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::67874928_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362491, took 20 ms
Event 67874928_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362428 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event 67874928_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 67874928_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 67874928_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
81 ms for event 17a5d043_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 67874928_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 67874928_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 67874928_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 67874928_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 67874928_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 67874928_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::67874928_TASK
for cluster PinotCluster, took 156 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 17a5d043_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 17a5d043_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 17a5d043_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 17a5d043_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 17a5d043_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 17a5d043_DEFAULT : Sending Message b328aa9a-5c5d-4cfc-b9b8-993f4cc3fe5d to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__23__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 17a5d043_DEFAULT : Sending Message febdd4c1-6f85-49dd-bb99-4d74a894e845 to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__23__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 8 ms for event 17a5d043_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 17a5d043_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 17a5d043_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::17a5d043_DEFAULT
for cluster PinotCluster, took 295 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 207 ms
TotalProcessed time for event: MessageChange took: 295 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 67874928_DEFAULT. Event session ID: 3000a812cfc0036
Event 67874928_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 67874928_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 67874928_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 67874928_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::67874928_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f26340fd_TASK. Event session ID: 3000a812cfc0036
Event f26340fd_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f26340fd_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f26340fd_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f26340fd_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::67874928_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::f26340fd_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f26340fd_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/b328aa9a-5c5d-4cfc-b9b8-
993f4cc3fe5d=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
Event PinotCluster::DEFAULT::67874928_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::67874928_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362648, took 27 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 1 ms.
Event 67874928_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::67874928_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::67874928_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362676, took 0 ms
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event 67874928_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362622 took 60 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 60 ms
for event 67874928_DEFAULT
Event PinotCluster::TASK::f26340fd_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::f26340fd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362651, took 31 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 67874928_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67874928_DEFAULT
Event PinotCluster::TASK::f26340fd_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f26340fd_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362682, took 20 ms
Event f26340fd_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362624 took 79 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 79 ms
for event f26340fd_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event f26340fd_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event f26340fd_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 67874928_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 67874928_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67874928_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event f26340fd_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event f26340fd_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f26340fd_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f26340fd_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f26340fd_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f26340fd_TASK
END: Invoking TASK controller pipeline for event MessageChange::f26340fd_TASK for
cluster PinotCluster, took 151 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8e402794_TASK. Event session ID: 3000a812cfc0036
Event 8e402794_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 8e402794_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8e402794_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8e402794_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::8e402794_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8e402794_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
27 ms for event 67874928_DEFAULT
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 33 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 67874928_DEFAULT
Event PinotCluster::TASK::8e402794_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8e402794_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362801, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 67874928_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67874928_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 67874928_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67874928_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 67874928_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67874928_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67874928_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::67874928_DEFAULT for cluster PinotCluster, took 217 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 453 ms
TotalProcessed time for event: IdealStateChange took: 217 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 297ed0d7_DEFAULT. Event session ID: 3000a812cfc0036
Event 297ed0d7_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 297ed0d7_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 297ed0d7_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 297ed0d7_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::8e402794_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8e402794_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362826, took 21 ms
Event 8e402794_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362775 took 73 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 73 ms
for event 8e402794_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 8e402794_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8e402794_TASK
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362865, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 297ed0d7_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::297ed0d7_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110362890, took 0 ms
Event 297ed0d7_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362840 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event 297ed0d7_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 297ed0d7_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 297ed0d7_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 8e402794_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8e402794_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8e402794_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8e402794_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8e402794_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8e402794_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::8e402794_TASK
for cluster PinotCluster, took 146 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 297ed0d7_TASK. Event session ID: 3000a812cfc0036
Event 297ed0d7_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 297ed0d7_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 297ed0d7_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 297ed0d7_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::297ed0d7_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::297ed0d7_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::297ed0d7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::297ed0d7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362946, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 297ed0d7_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 297ed0d7_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 297ed0d7_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 297ed0d7_DEFAULT
Event PinotCluster::TASK::297ed0d7_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::297ed0d7_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110362970, took 34 ms
Event 297ed0d7_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110362921 took 84 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 84 ms
for event 297ed0d7_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 297ed0d7_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 297ed0d7_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 297ed0d7_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 297ed0d7_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 297ed0d7_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 297ed0d7_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 297ed0d7_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 297ed0d7_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 297ed0d7_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 297ed0d7_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::297ed0d7_DEFAULT
for cluster PinotCluster, took 207 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 159 ms
TotalProcessed time for event: MessageChange took: 207 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8e402794_DEFAULT. Event session ID: 3000a812cfc0036
Event 8e402794_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 8e402794_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 8e402794_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 8e402794_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::8e402794_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::8e402794_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 297ed0d7_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 297ed0d7_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 297ed0d7_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 297ed0d7_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 297ed0d7_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 297ed0d7_TASK
END: Invoking TASK controller pipeline for event MessageChange::297ed0d7_TASK for
cluster PinotCluster, took 154 ms
Event PinotCluster::DEFAULT::8e402794_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8e402794_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110363072, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 8e402794_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::8e402794_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8e402794_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110363096, took 0 ms
Event 8e402794_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110363047 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 8e402794_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 8e402794_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8e402794_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 8e402794_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 8e402794_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8e402794_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8e402794_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::ExternalViewComputeStage
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
44 ms for event 8e402794_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 69 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 8e402794_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 8e402794_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8e402794_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 8e402794_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8e402794_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 8e402794_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8e402794_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8e402794_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::8e402794_DEFAULT for cluster PinotCluster, took 226 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 374 ms
TotalProcessed time for event: CurrentStateChange took: 226 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 29cceac2_TASK. Event session ID: 3000a812cfc0036
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 29cceac2_DEFAULT. Event session ID: 3000a812cfc0036
Event 29cceac2_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 29cceac2_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 29cceac2_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 29cceac2_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event 29cceac2_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 29cceac2_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 29cceac2_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 29cceac2_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::29cceac2_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::29cceac2_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/9e2eea3f-df1f-4c9c-a67e-
16333cbbcf63=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/2035cad6-146c-4b60-93a3-
e7ed17557d52=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/f41c28fb-0fc6-466f-8772-
5761240bf5a3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/9e2eea3f-df1f-4c9c-a67e-
16333cbbcf63=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/2035cad6-146c-4b60-93a3-
e7ed17557d52=-101,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/f41c28fb-0fc6-466f-8772-
5761240bf5a3=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::29cceac2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::29cceac2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364609, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110364609, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 29cceac2_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::29cceac2_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110364639, took 0 ms
Event 29cceac2_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110364580 took 65 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 65 ms
for event 29cceac2_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::TASK::29cceac2_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::29cceac2_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364635, took 24 ms
Event 29cceac2_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110364580 took 80 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 80 ms
for event 29cceac2_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event 29cceac2_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29cceac2_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 29cceac2_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29cceac2_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 29cceac2_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 29cceac2_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 29cceac2_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 29cceac2_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 29cceac2_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 29cceac2_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 29cceac2_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29cceac2_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29cceac2_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::29cceac2_TASK for
cluster PinotCluster, took 152 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 81b2be78_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 29cceac2_DEFAULT
Event PinotCluster::TASK::81b2be78_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::81b2be78_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event 81b2be78_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 81b2be78_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 81b2be78_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::81b2be78_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::81b2be78_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 29cceac2_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 29cceac2_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 29cceac2_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 29cceac2_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29cceac2_DEFAULT
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::81b2be78_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::81b2be78_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364772, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 29cceac2_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29cceac2_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 29cceac2_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::29cceac2_DEFAULT
for cluster PinotCluster, took 222 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 222 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 81b2be78_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event 81b2be78_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 81b2be78_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 81b2be78_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::81b2be78_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::81b2be78_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364797, took 22 ms
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
Event 81b2be78_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110364732 took 88 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 88 ms
for event 81b2be78_TASK
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 81b2be78_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 81b2be78_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110364842, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 81b2be78_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::81b2be78_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110364866, took 0 ms
Event 81b2be78_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110364802 took 69 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 69 ms
for event 81b2be78_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
18 ms for event 81b2be78_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
50 ms for event 81b2be78_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 81b2be78_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 81b2be78_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 81b2be78_DEFAULT
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 81b2be78_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 81b2be78_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 81b2be78_TASK
END: Invoking TASK controller pipeline for event MessageChange::81b2be78_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e645be2a_TASK. Event session ID: 3000a812cfc0036
Event e645be2a_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e645be2a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e645be2a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e645be2a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::e645be2a_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e645be2a_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::e645be2a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e645be2a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364914, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 58 ms for event 81b2be78_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 81b2be78_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 81b2be78_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 81b2be78_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::e645be2a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e645be2a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110364939, took 21 ms
Event e645be2a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110364889 took 71 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event e645be2a_TASK
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 31
ms for event e645be2a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e645be2a_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
93 ms for event 81b2be78_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
61 ms for event e645be2a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e645be2a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e645be2a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e645be2a_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e645be2a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e645be2a_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::e645be2a_TASK
for cluster PinotCluster, took 164 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 81b2be78_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 81b2be78_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 81b2be78_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 81b2be78_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 81b2be78_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 81b2be78_DEFAULT : Sending Message 74381203-0104-47ec-816f-a2f69da517a3 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__35__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 81b2be78_DEFAULT : Sending Message a9e9a512-75e8-4b3c-bbcb-255c01a2f383 to
Server_pinot-server-10.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__35__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 10 ms for event 81b2be78_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 81b2be78_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 81b2be78_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::81b2be78_DEFAULT
for cluster PinotCluster, took 297 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 196 ms
TotalProcessed time for event: MessageChange took: 297 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e645be2a_DEFAULT. Event session ID: 3000a812cfc0036
Event e645be2a_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event e645be2a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e645be2a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e645be2a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: f16b931d_TASK. Event session ID: 3000a812cfc0036
Event f16b931d_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event f16b931d_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event f16b931d_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event f16b931d_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::f16b931d_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::f16b931d_TASK : Refreshed 1 property ClusterConstraint
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 25
ms.
Start to refresh stale message cache
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/74381203-0104-47ec-816f-
a2f69da517a3=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 28
ms.
Start to refresh stale message cache
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 3ms
103 START: CallbackHandler 48, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 48, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0041
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 2ms
103 START: CallbackHandler 57, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 57 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler57, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 57, /PinotCluster/INSTANCES/Server_pinot-server-
10.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 7ms
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365131, took 39 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e645be2a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e645be2a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365170, took 0 ms
Event e645be2a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365099 took 77 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 78 ms
for event e645be2a_DEFAULT
Event PinotCluster::TASK::f16b931d_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::f16b931d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365136, took 41 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event e645be2a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e645be2a_DEFAULT
Event PinotCluster::TASK::f16b931d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::f16b931d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365177, took 23 ms
Event f16b931d_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365102 took 99 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 99 ms
for event f16b931d_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event f16b931d_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f16b931d_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 58 ms for event e645be2a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e645be2a_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e645be2a_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 1
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
26 ms for event e645be2a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
61 ms for event f16b931d_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f16b931d_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event f16b931d_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event f16b931d_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f16b931d_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event f16b931d_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::f16b931d_TASK for
cluster PinotCluster, took 182 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8a8e72ab_TASK. Event session ID: 3000a812cfc0036
Event 8a8e72ab_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 8a8e72ab_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8a8e72ab_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8a8e72ab_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::8a8e72ab_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8a8e72ab_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event e645be2a_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event e645be2a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e645be2a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event e645be2a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e645be2a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event PinotCluster::TASK::8a8e72ab_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8a8e72ab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365309, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event e645be2a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e645be2a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e645be2a_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::e645be2a_DEFAULT for cluster PinotCluster, took 236 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 446 ms
TotalProcessed time for event: IdealStateChange took: 236 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 09bcbfde_DEFAULT. Event session ID: 3000a812cfc0036
Event 09bcbfde_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 09bcbfde_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 09bcbfde_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 09bcbfde_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::8a8e72ab_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8a8e72ab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365334, took 37 ms
Event 8a8e72ab_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365284 took 88 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 88 ms
for event 8a8e72ab_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 35
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 8a8e72ab_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8a8e72ab_TASK
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365377, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 09bcbfde_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::09bcbfde_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365402, took 0 ms
Event 09bcbfde_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365335 took 73 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 73 ms
for event 09bcbfde_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 09bcbfde_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 09bcbfde_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 8a8e72ab_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8a8e72ab_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8a8e72ab_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8a8e72ab_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8a8e72ab_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8a8e72ab_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event CurrentStateChange::8a8e72ab_TASK
for cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 09bcbfde_TASK. Event session ID: 3000a812cfc0036
Event 09bcbfde_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 09bcbfde_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 09bcbfde_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 09bcbfde_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::09bcbfde_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::09bcbfde_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 09bcbfde_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 09bcbfde_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 09bcbfde_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Event PinotCluster::TASK::09bcbfde_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::09bcbfde_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365471, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 09bcbfde_DEFAULT
Event PinotCluster::TASK::09bcbfde_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::09bcbfde_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110365501, took 20 ms
Event 09bcbfde_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365445 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event 09bcbfde_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 09bcbfde_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 09bcbfde_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 09bcbfde_TASK
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 09bcbfde_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 09bcbfde_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 09bcbfde_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 09bcbfde_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 09bcbfde_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 09bcbfde_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 09bcbfde_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::09bcbfde_DEFAULT
for cluster PinotCluster, took 224 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 175 ms
TotalProcessed time for event: MessageChange took: 224 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 8a8e72ab_DEFAULT. Event session ID: 3000a812cfc0036
Event 8a8e72ab_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 8a8e72ab_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 8a8e72ab_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 8a8e72ab_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 09bcbfde_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 09bcbfde_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 09bcbfde_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 09bcbfde_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 09bcbfde_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 09bcbfde_TASK
END: Invoking TASK controller pipeline for event MessageChange::09bcbfde_TASK for
cluster PinotCluster, took 148 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365585, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 8a8e72ab_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8a8e72ab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110365610, took 0 ms
Event 8a8e72ab_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110365559 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event 8a8e72ab_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 8a8e72ab_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event 8a8e72ab_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 8a8e72ab_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 8a8e72ab_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8a8e72ab_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8a8e72ab_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 23 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 8a8e72ab_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 8a8e72ab_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 59 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 8a8e72ab_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 8a8e72ab_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 8a8e72ab_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8a8e72ab_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 8a8e72ab_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8a8e72ab_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8a8e72ab_DEFAULT
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::8a8e72ab_DEFAULT for cluster PinotCluster, took 212 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 404 ms
TotalProcessed time for event: CurrentStateChange took: 212 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 022b931e_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 022b931e_TASK. Event session ID: 3000a812cfc0036
Event 022b931e_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 022b931e_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 022b931e_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 022b931e_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 022b931e_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 022b931e_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 022b931e_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event 022b931e_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::022b931e_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
Event PinotCluster::DEFAULT::022b931e_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::022b931e_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::022b931e_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/8a3461f0-4f8e-41af-911d-
6b6ba4618ebe=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/b5b7c1a1-7cfa-4d95-9699-
e9b362d05537=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/b5b7c1a1-7cfa-4d95-9699-
e9b362d05537=-101}
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::022b931e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::022b931e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366196, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::022b931e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::022b931e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366196, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 022b931e_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::022b931e_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::022b931e_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366225, took 0 ms
Event 022b931e_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366167 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event 022b931e_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::022b931e_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::022b931e_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366221, took 24 ms
Event 022b931e_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366167 took 78 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 78 ms
for event 022b931e_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 022b931e_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 022b931e_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 022b931e_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 022b931e_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 022b931e_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 022b931e_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 022b931e_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 022b931e_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 022b931e_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 022b931e_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 022b931e_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 022b931e_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 022b931e_TASK
END: Invoking TASK controller pipeline for event MessageChange::022b931e_TASK for
cluster PinotCluster, took 151 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1ad2803c_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 022b931e_DEFAULT
Event PinotCluster::TASK::1ad2803c_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::1ad2803c_TASK : Refreshed 11 property IdealState took 14
ms. Selective: true
Event 1ad2803c_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 1ad2803c_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 1ad2803c_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::1ad2803c_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::1ad2803c_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 022b931e_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 022b931e_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 022b931e_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 022b931e_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 022b931e_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 022b931e_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 022b931e_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 022b931e_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event MessageChange::022b931e_DEFAULT
for cluster PinotCluster, took 214 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 214 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 1ad2803c_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::1ad2803c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1ad2803c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366358, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : Refreshed 11 property IdealState
took 15 ms. Selective: true
Event 1ad2803c_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 1ad2803c_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 1ad2803c_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::1ad2803c_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::1ad2803c_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366383, took 23 ms
Event 1ad2803c_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366318 took 88 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 88 ms
for event 1ad2803c_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 1ad2803c_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 1ad2803c_TASK
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366423, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 1ad2803c_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::1ad2803c_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366448, took 0 ms
Event 1ad2803c_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366381 took 72 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 72 ms
for event 1ad2803c_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 1ad2803c_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ad2803c_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event 1ad2803c_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1ad2803c_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 1ad2803c_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 1ad2803c_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1ad2803c_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 1ad2803c_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::1ad2803c_TASK for
cluster PinotCluster, took 164 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 2faec7b0_TASK. Event session ID: 3000a812cfc0036
Event 2faec7b0_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 2faec7b0_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 2faec7b0_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 2faec7b0_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::2faec7b0_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::2faec7b0_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 1ad2803c_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 1ad2803c_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ad2803c_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event 1ad2803c_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::2faec7b0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2faec7b0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366508, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Computing BestPossibleMapping for brokerResource
Start calculating the new best possible assignment.
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Event PinotCluster::TASK::2faec7b0_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::2faec7b0_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366538, took 20 ms
Event 2faec7b0_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366482 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event 2faec7b0_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 32
ms for event 2faec7b0_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2faec7b0_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
95 ms for event 1ad2803c_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event 2faec7b0_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2faec7b0_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 2faec7b0_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 2faec7b0_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2faec7b0_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 2faec7b0_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::2faec7b0_TASK
for cluster PinotCluster, took 165 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 1ad2803c_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 1ad2803c_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ad2803c_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 1ad2803c_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ad2803c_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 1ad2803c_DEFAULT : Sending Message 771989f7-31b1-457a-b4e8-dda5477c55b9 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__12__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 1ad2803c_DEFAULT : Sending Message 3951d428-a91f-4d4d-a915-7a103a3a8735 to
Server_pinot-server-8.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__12__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 1ad2803c_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 1ad2803c_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 1ad2803c_DEFAULT
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::1ad2803c_DEFAULT
for cluster PinotCluster, took 298 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 189 ms
TotalProcessed time for event: MessageChange took: 298 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 2faec7b0_DEFAULT. Event session ID: 3000a812cfc0036
Event 2faec7b0_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 2faec7b0_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 2faec7b0_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 2faec7b0_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: e10fc544_TASK. Event session ID: 3000a812cfc0036
Event e10fc544_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e10fc544_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e10fc544_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e10fc544_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
Event PinotCluster::TASK::e10fc544_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e10fc544_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 40, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 40, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0033 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 67, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 67 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler67, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 67, /PinotCluster/INSTANCES/Server_pinot-server-
8.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366706, took 29 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 2faec7b0_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::2faec7b0_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366735, took 0 ms
Event 2faec7b0_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366679 took 62 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 62 ms
for event 2faec7b0_DEFAULT
Event PinotCluster::TASK::e10fc544_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::e10fc544_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366709, took 32 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::e10fc544_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e10fc544_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366741, took 21 ms
Event e10fc544_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366682 took 81 for TASK pipeline
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event 2faec7b0_DEFAULT
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event e10fc544_TASK
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2faec7b0_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event e10fc544_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event e10fc544_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event 2faec7b0_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 2faec7b0_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2faec7b0_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event e10fc544_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e10fc544_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e10fc544_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e10fc544_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e10fc544_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e10fc544_TASK
END: Invoking TASK controller pipeline for event MessageChange::e10fc544_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: e20582bc_TASK. Event session ID: 3000a812cfc0036
Event e20582bc_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e20582bc_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e20582bc_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e20582bc_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Event PinotCluster::TASK::e20582bc_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e20582bc_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 2faec7b0_DEFAULT
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 33
ms for event 2faec7b0_DEFAULT
Event PinotCluster::TASK::e20582bc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e20582bc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366865, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 2faec7b0_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2faec7b0_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 2faec7b0_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2faec7b0_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 2faec7b0_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 2faec7b0_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 2faec7b0_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::2faec7b0_DEFAULT for cluster PinotCluster, took 223 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 438 ms
TotalProcessed time for event: IdealStateChange took: 223 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a8826f45_DEFAULT. Event session ID: 3000a812cfc0036
Event a8826f45_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event a8826f45_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event a8826f45_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event a8826f45_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::e20582bc_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e20582bc_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110366890, took 20 ms
Event e20582bc_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366839 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event e20582bc_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event e20582bc_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e20582bc_TASK
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366927, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event a8826f45_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::a8826f45_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110366952, took 0 ms
Event a8826f45_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366902 took 56 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 56 ms
for event a8826f45_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
21 ms for event a8826f45_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a8826f45_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event e20582bc_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e20582bc_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 1 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event e20582bc_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e20582bc_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e20582bc_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e20582bc_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::e20582bc_TASK
for cluster PinotCluster, took 146 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: a8826f45_TASK. Event session ID: 3000a812cfc0036
Event a8826f45_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event a8826f45_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event a8826f45_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event a8826f45_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::a8826f45_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::a8826f45_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::a8826f45_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a8826f45_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110367010, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event a8826f45_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event a8826f45_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a8826f45_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::a8826f45_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::a8826f45_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110367034, took 25 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event a8826f45_DEFAULT
Event a8826f45_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110366985 took 75 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 75 ms
for event a8826f45_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event a8826f45_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event a8826f45_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event a8826f45_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event a8826f45_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a8826f45_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event a8826f45_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a8826f45_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event a8826f45_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event a8826f45_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event a8826f45_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::a8826f45_DEFAULT
for cluster PinotCluster, took 213 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 170 ms
TotalProcessed time for event: MessageChange took: 213 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: e20582bc_DEFAULT. Event session ID: 3000a812cfc0036
Event e20582bc_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event e20582bc_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e20582bc_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e20582bc_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event a8826f45_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a8826f45_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event a8826f45_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event a8826f45_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a8826f45_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event a8826f45_TASK
END: Invoking TASK controller pipeline for event MessageChange::a8826f45_TASK for
cluster PinotCluster, took 149 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110367141, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e20582bc_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e20582bc_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110367166, took 0 ms
Event e20582bc_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110367115 took 57 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 57 ms
for event e20582bc_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
22 ms for event e20582bc_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e20582bc_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event e20582bc_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e20582bc_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e20582bc_DEFAULT
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e20582bc_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event e20582bc_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event e20582bc_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 62 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event e20582bc_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e20582bc_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event e20582bc_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e20582bc_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event e20582bc_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e20582bc_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e20582bc_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::e20582bc_DEFAULT for cluster PinotCluster, took 216 ms
Callback time for event: CurrentStateChange took: 1 ms
InQueue time for event: CurrentStateChange took: 390 ms
TotalProcessed time for event: CurrentStateChange took: 216 ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 6abeea94_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 6abeea94_TASK. Event session ID: 3000a812cfc0036
Event 6abeea94_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 6abeea94_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 6abeea94_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 6abeea94_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 6abeea94_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 6abeea94_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event 6abeea94_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 6abeea94_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
Event PinotCluster::TASK::6abeea94_TASK : Refreshed 9 property StateModelDefinition
took 4 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::6abeea94_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/22d49988-66a2-49bb-849c-
cda582a5f789=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e7e8440b-079d-4205-90dc-
ecf2eb374d26=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/b4583951-c0ac-4ec7-83b0-
516154921d1f=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES/22d49988-66a2-49bb-849c-
cda582a5f789=-101,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/e7e8440b-079d-4205-90dc-
ecf2eb374d26=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/b4583951-c0ac-4ec7-83b0-
516154921d1f=-101}
END: InstanceMessagesCache.refresh(), 3 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::6abeea94_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::6abeea94_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110368815, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110368815, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 6abeea94_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::6abeea94_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110368843, took 0 ms
Event 6abeea94_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110368788 took 61 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 61 ms
for event 6abeea94_DEFAULT
Event PinotCluster::TASK::6abeea94_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::6abeea94_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110368839, took 25 ms
Event 6abeea94_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110368788 took 77 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 77 ms
for event 6abeea94_TASK
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 4ms
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 6abeea94_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6abeea94_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 6abeea94_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 6abeea94_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 54 ms for event 6abeea94_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 6abeea94_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6abeea94_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Start calculating the new best possible assignment.
Finish computing new ideal states for resources: []
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 6abeea94_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 6abeea94_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 6abeea94_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 6abeea94_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 6abeea94_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 6abeea94_TASK
END: Invoking TASK controller pipeline for event MessageChange::6abeea94_TASK for
cluster PinotCluster, took 150 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 8de5f67a_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
25 ms for event 6abeea94_DEFAULT
Event PinotCluster::TASK::8de5f67a_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::8de5f67a_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event 8de5f67a_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 8de5f67a_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 8de5f67a_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::8de5f67a_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::8de5f67a_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 35
ms for event 6abeea94_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 6abeea94_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6abeea94_DEFAULT
Event PinotCluster::TASK::8de5f67a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8de5f67a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110368979, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 8 ms
for event 6abeea94_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6abeea94_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 6abeea94_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 6abeea94_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 6abeea94_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::6abeea94_DEFAULT
for cluster PinotCluster, took 224 ms
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Callback time for event: MessageChange took: 0 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 224 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 8de5f67a_DEFAULT. Event session ID: 3000a812cfc0036
Event PinotCluster::TASK::8de5f67a_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::8de5f67a_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369004, took 21 ms
Event 8de5f67a_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110368938 took 88 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 88 ms
for event 8de5f67a_TASK
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : Refreshed 11 property IdealState
took 16 ms. Selective: true
Event 8de5f67a_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 8de5f67a_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 8de5f67a_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 8de5f67a_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8de5f67a_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369053, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 8de5f67a_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::8de5f67a_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369078, took 0 ms
Event 8de5f67a_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369012 took 70 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 71 ms
for event 8de5f67a_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 8de5f67a_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8de5f67a_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 8de5f67a_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 8de5f67a_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8de5f67a_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 8de5f67a_TASK
END: Invoking TASK controller pipeline for event MessageChange::8de5f67a_TASK for
cluster PinotCluster, took 161 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e585641d_TASK. Event session ID: 3000a812cfc0036
Event e585641d_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event e585641d_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event e585641d_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event e585641d_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::e585641d_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 8de5f67a_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8de5f67a_DEFAULT
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::e585641d_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 22
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::e585641d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e585641d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369128, took 26 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 8de5f67a_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 8de5f67a_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 8de5f67a_DEFAULT
Start computing new ideal states for resources: []
Event 8de5f67a_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Event PinotCluster::TASK::e585641d_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::e585641d_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369154, took 20 ms
Event e585641d_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369099 took 76 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 76 ms
for event e585641d_TASK
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event e585641d_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e585641d_TASK
Computing BestPossibleMapping for clickstreamd_REALTIME
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event e585641d_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e585641d_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event e585641d_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event e585641d_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e585641d_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event e585641d_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::e585641d_TASK
for cluster PinotCluster, took 150 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
91 ms for event 8de5f67a_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 34
ms for event 8de5f67a_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 13
ms for event 8de5f67a_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 8de5f67a_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 8de5f67a_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8de5f67a_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event 8de5f67a_DEFAULT : Sending Message ddb902a1-2566-43d0-ab8b-7d06a296b6f5 to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__4__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event 8de5f67a_DEFAULT : Sending Message f578c163-0eaf-47e6-835b-910cebf671bb to
Server_pinot-server-6.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__4__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event 8de5f67a_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 8de5f67a_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 8de5f67a_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event MessageChange::8de5f67a_DEFAULT
for cluster PinotCluster, took 303 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 198 ms
TotalProcessed time for event: MessageChange took: 303 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: e585641d_DEFAULT. Event session ID: 3000a812cfc0036
Event e585641d_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event e585641d_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event e585641d_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event e585641d_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::e585641d_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: c757a97f_TASK. Event session ID: 3000a812cfc0036
Event c757a97f_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event c757a97f_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event c757a97f_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event c757a97f_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::e585641d_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
Event PinotCluster::TASK::c757a97f_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::c757a97f_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 21
ms.
Start to refresh stale message cache
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 36, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 36, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/3000a812cfc0030 listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 73, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 73 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler73, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 4
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 73, /PinotCluster/INSTANCES/Server_pinot-server-
6.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 5ms
Event PinotCluster::DEFAULT::e585641d_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e585641d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369342, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event e585641d_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::e585641d_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::e585641d_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369372, took 0 ms
Event e585641d_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369315 took 64 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 64 ms
for event e585641d_DEFAULT
Event PinotCluster::TASK::c757a97f_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::c757a97f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369345, took 34 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::TASK::c757a97f_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::c757a97f_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369379, took 19 ms
Event c757a97f_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369318 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event c757a97f_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event e585641d_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 1
ms for event e585641d_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 33
ms for event c757a97f_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c757a97f_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 67 ms for event e585641d_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event e585641d_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
4 ms for event e585641d_DEFAULT
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
55 ms for event c757a97f_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c757a97f_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event c757a97f_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event c757a97f_TASK
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c757a97f_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event c757a97f_TASK
END: Invoking TASK controller pipeline for event MessageChange::c757a97f_TASK for
cluster PinotCluster, took 169 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 67d6d598_TASK. Event session ID: 3000a812cfc0036
Event 67d6d598_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 67d6d598_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 67d6d598_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 67d6d598_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::67d6d598_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::67d6d598_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event e585641d_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 31
ms for event e585641d_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event e585641d_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e585641d_DEFAULT
Event PinotCluster::TASK::67d6d598_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::67d6d598_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369516, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event e585641d_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e585641d_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event e585641d_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event e585641d_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event e585641d_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::e585641d_DEFAULT for cluster PinotCluster, took 233 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 448 ms
TotalProcessed time for event: IdealStateChange took: 233 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: fb7bfe59_DEFAULT. Event session ID: 3000a812cfc0036
Event fb7bfe59_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event fb7bfe59_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event fb7bfe59_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event fb7bfe59_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::67d6d598_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::67d6d598_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369540, took 20 ms
Event 67d6d598_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369487 took 74 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 74 ms
for event 67d6d598_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 20
ms for event 67d6d598_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 67d6d598_TASK
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369573, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event fb7bfe59_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::fb7bfe59_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369597, took 0 ms
Event fb7bfe59_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369548 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event fb7bfe59_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
30 ms for event fb7bfe59_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event fb7bfe59_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
63 ms for event 67d6d598_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 67d6d598_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 67d6d598_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 67d6d598_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 67d6d598_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 67d6d598_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::67d6d598_TASK
for cluster PinotCluster, took 159 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: fb7bfe59_TASK. Event session ID: 3000a812cfc0036
Event fb7bfe59_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event fb7bfe59_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event fb7bfe59_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event fb7bfe59_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::fb7bfe59_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::fb7bfe59_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 57 ms for event fb7bfe59_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event fb7bfe59_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fb7bfe59_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::fb7bfe59_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fb7bfe59_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369672, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::fb7bfe59_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::fb7bfe59_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110369697, took 20 ms
Event fb7bfe59_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369646 took 72 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 72 ms
for event fb7bfe59_TASK
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
32 ms for event fb7bfe59_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event fb7bfe59_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event fb7bfe59_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event fb7bfe59_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event fb7bfe59_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fb7bfe59_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event fb7bfe59_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event fb7bfe59_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event fb7bfe59_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event fb7bfe59_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event fb7bfe59_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::fb7bfe59_DEFAULT
for cluster PinotCluster, took 227 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: MessageChange took: 4 ms
InQueue time for event: MessageChange took: 179 ms
TotalProcessed time for event: MessageChange took: 227 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 67d6d598_DEFAULT. Event session ID: 3000a812cfc0036
Event 67d6d598_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 67d6d598_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 67d6d598_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 67d6d598_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event fb7bfe59_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fb7bfe59_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event fb7bfe59_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event fb7bfe59_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fb7bfe59_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event fb7bfe59_TASK
END: Invoking TASK controller pipeline for event MessageChange::fb7bfe59_TASK for
cluster PinotCluster, took 145 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369800, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 67d6d598_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::67d6d598_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110369824, took 0 ms
Event 67d6d598_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110369775 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 67d6d598_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 67d6d598_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67d6d598_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 56 ms for event 67d6d598_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 67d6d598_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67d6d598_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67d6d598_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 67d6d598_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 67d6d598_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 59 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 67d6d598_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67d6d598_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 7 ms
for event 67d6d598_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67d6d598_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 4 ms for event 67d6d598_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 67d6d598_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 67d6d598_DEFAULT
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::67d6d598_DEFAULT for cluster PinotCluster, took 207 ms
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 413 ms
TotalProcessed time for event: CurrentStateChange took: 207 ms
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 4337b2eb_DEFAULT. Event session ID: 3000a812cfc0036
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 4337b2eb_TASK. Event session ID: 3000a812cfc0036
Event 4337b2eb_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 4337b2eb_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 4337b2eb_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 4337b2eb_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 4337b2eb_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 4337b2eb_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 4337b2eb_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event 4337b2eb_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::TASK::4337b2eb_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::4337b2eb_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
START: InstanceMessagesCache.refresh()
START: InstanceMessagesCache.refresh()
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
103 START: CallbackHandler 71, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 71 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler71, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 71,
/PinotCluster/INSTANCES/Broker_10.5.228.237_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 61, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 61 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler61, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES took: 0
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 61,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
103 START: CallbackHandler 70, INVOKE
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 70 subscribing changes listener to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES, callback type: CALLBACK,
event types: [NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler70, Subscribing to path:
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 70,
/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/96f1f147-15a6-4021-92c2-
7b3ccf71f793=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/fa41010e-4f0c-4652-80aa-
9da67370ed14=-101}
Fail to read record for paths:
{/PinotCluster/INSTANCES/Broker_10.5.226.104_8099/MESSAGES/96f1f147-15a6-4021-92c2-
7b3ccf71f793=-101,
/PinotCluster/INSTANCES/Broker_10.5.234.128_8099/MESSAGES/fa41010e-4f0c-4652-80aa-
9da67370ed14=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374462, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 4337b2eb_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::4337b2eb_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374492, took 0 ms
Event PinotCluster::TASK::4337b2eb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4337b2eb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374462, took 30 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 4337b2eb_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374432 took 66 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 66 ms
for event 4337b2eb_DEFAULT
103 START: CallbackHandler 96, INVOKE /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: Generic GenericClusterController.onIdealStateChange() for cluster
PinotCluster
Controller stopping periodical rebalance timer at period 9223372036854775807
END: GenericClusterController.onIdealStateChange() for cluster PinotCluster
103 END:INVOKE CallbackHandler 96, /PinotCluster/IDEALSTATES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 1ms
Event PinotCluster::TASK::4337b2eb_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::4337b2eb_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374492, took 20 ms
Event 4337b2eb_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374432 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event 4337b2eb_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 4337b2eb_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4337b2eb_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 4337b2eb_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4337b2eb_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event 4337b2eb_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 4337b2eb_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4337b2eb_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
56 ms for event 4337b2eb_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4337b2eb_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 4337b2eb_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 4337b2eb_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4337b2eb_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 4337b2eb_TASK
END: Invoking TASK controller pipeline for event MessageChange::4337b2eb_TASK for
cluster PinotCluster, took 156 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b6d02cab_TASK. Event session ID: 3000a812cfc0036
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 4337b2eb_DEFAULT
Event PinotCluster::TASK::b6d02cab_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::b6d02cab_TASK : Refreshed 11 property IdealState took 15
ms. Selective: true
Event b6d02cab_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event b6d02cab_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event b6d02cab_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::b6d02cab_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::b6d02cab_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 29
ms for event 4337b2eb_DEFAULT
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 4337b2eb_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4337b2eb_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 4337b2eb_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4337b2eb_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 4337b2eb_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 4337b2eb_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 4337b2eb_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::4337b2eb_DEFAULT
for cluster PinotCluster, took 214 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 1 ms
TotalProcessed time for event: MessageChange took: 214 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: b6d02cab_DEFAULT. Event session ID: 3000a812cfc0036
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Event PinotCluster::TASK::b6d02cab_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b6d02cab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374628, took 25 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : Refreshed 11 property IdealState
took 14 ms. Selective: true
Event b6d02cab_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event b6d02cab_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event b6d02cab_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::b6d02cab_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::b6d02cab_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374653, took 20 ms
Event b6d02cab_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374588 took 86 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 86 ms
for event b6d02cab_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 18
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event b6d02cab_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b6d02cab_TASK
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374684, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b6d02cab_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::b6d02cab_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374708, took 0 ms
Event b6d02cab_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374646 took 67 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 67 ms
for event b6d02cab_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event b6d02cab_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b6d02cab_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event b6d02cab_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b6d02cab_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event b6d02cab_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event b6d02cab_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b6d02cab_TASK
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event b6d02cab_TASK
END: Invoking TASK controller pipeline for event MessageChange::b6d02cab_TASK for
cluster PinotCluster, took 157 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 7a2b6143_TASK. Event session ID: 3000a812cfc0036
Event 7a2b6143_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 7a2b6143_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 7a2b6143_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 7a2b6143_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::7a2b6143_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::7a2b6143_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 53 ms for event b6d02cab_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event b6d02cab_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b6d02cab_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Event PinotCluster::TASK::7a2b6143_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7a2b6143_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374769, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event b6d02cab_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.
Start emergency rebalance.
Start calculating the new baseline.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Computing BestPossibleMapping for brokerResource
Computing BestPossibleMapping for tradesntt_REALTIME
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
Global baseline calculation completed and has been persisted into metadata store.
Computing BestPossibleMapping for amxordersntt_REALTIME
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
Event PinotCluster::TASK::7a2b6143_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::7a2b6143_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374792, took 20 ms
Event 7a2b6143_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374745 took 68 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 68 ms
for event 7a2b6143_TASK
Computing BestPossibleMapping for clickstreamweb_REALTIME
Computing BestPossibleMapping for clickstream_REALTIME
Computing BestPossibleMapping for positionamxgpx_REALTIME
Computing BestPossibleMapping for clickstreamd_REALTIME
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 17
ms for event 7a2b6143_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 7a2b6143_TASK
Computing BestPossibleMapping for amxordersgpx_REALTIME
Computing IdealState for leadControllerResource
Found the following participants with tag controller for leadControllerResource:
instances: [Controller_pinot-controller_9000, Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000, Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000], liveEnabledInstances: [Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000]
Computing BestPossibleMapping for clientlogin_REALTIME
Computing BestPossibleMapping for positionamxntt_REALTIME
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
81 ms for event b6d02cab_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
51 ms for event 7a2b6143_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 7a2b6143_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 7a2b6143_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 7a2b6143_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7a2b6143_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 7a2b6143_TASK
END: Invoking TASK controller pipeline for event IdealStateChange::7a2b6143_TASK
for cluster PinotCluster, took 138 ms
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event b6d02cab_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event b6d02cab_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b6d02cab_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event b6d02cab_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b6d02cab_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
Event b6d02cab_DEFAULT : Sending Message 89a30eb7-90cf-4baf-b6ba-ffb699212dac to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__16__883__20241028T1004Z|[] from:CONSUMING
to:ONLINE, relayMessages: 0
Event b6d02cab_DEFAULT : Sending Message 326d57f5-90de-4fca-9c29-65e0c182c3a1 to
Server_pinot-server-12.pinot-server.pinot.svc.cluster.local_8098 transit
clickstreamd_REALTIME.clickstreamd__16__884__20241028T1012Z|[] from:OFFLINE
to:CONSUMING, relayMessages: 0
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 9 ms for event b6d02cab_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event b6d02cab_DEFAULT
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event b6d02cab_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::b6d02cab_DEFAULT
for cluster PinotCluster, took 276 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 185 ms
TotalProcessed time for event: MessageChange took: 276 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
IdealStateChange, ID: 7a2b6143_DEFAULT. Event session ID: 3000a812cfc0036
Event 7a2b6143_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 7a2b6143_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 7a2b6143_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 7a2b6143_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 2
START: GenericClusterController.onMessage() for cluster PinotCluster
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 3ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 94f2a7fe_TASK. Event session ID: 3000a812cfc0036
Event 94f2a7fe_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 94f2a7fe_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 94f2a7fe_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 94f2a7fe_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
Event PinotCluster::TASK::94f2a7fe_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::94f2a7fe_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Fail to read record for paths: {/PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES/89a30eb7-90cf-4baf-b6ba-
ffb699212dac=-101}
END: InstanceMessagesCache.refresh(), 2 of Messages read from ZooKeeper. took 23
ms.
Start to refresh stale message cache
103 START: CallbackHandler 38, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
START: GenericClusterController.onStateChange()
END: GenericClusterController.onStateChange()
103 END:INVOKE CallbackHandler 38, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/CURRENTSTATES/1000a812ceb0048
listener: org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
Took: 1ms
103 START: CallbackHandler 56, INVOKE /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK
CallbackHandler 56 subscribing changes listener to path:
/PinotCluster/INSTANCES/Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098/MESSAGES, callback type: CALLBACK, event types:
[NodeChildrenChanged], listener:
org.apache.helix.controller.GenericHelixController@d542266, watchChild: false
CallbackHandler56, Subscribing to path: /PinotCluster/INSTANCES/Server_pinot-
server-12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES took: 1
START: GenericClusterController.onMessage() for cluster PinotCluster
END: GenericClusterController.onMessage() for cluster PinotCluster
103 END:INVOKE CallbackHandler 56, /PinotCluster/INSTANCES/Server_pinot-server-
12.pinot-server.pinot.svc.cluster.local_8098/MESSAGES listener:
org.apache.helix.controller.GenericHelixController@d542266 type: CALLBACK Took: 2ms
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : 1 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374952, took 28 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7a2b6143_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::7a2b6143_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110374980, took 0 ms
Event PinotCluster::TASK::94f2a7fe_TASK : 1 properties refreshed from ZK.
Event PinotCluster::TASK::94f2a7fe_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110374955, took 47 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 7a2b6143_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374922 took 81 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 81 ms
for event 7a2b6143_DEFAULT
Event PinotCluster::TASK::94f2a7fe_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::94f2a7fe_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110375002, took 20 ms
Event 94f2a7fe_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110374926 took 96 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 96 ms
for event 94f2a7fe_TASK
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 7a2b6143_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7a2b6143_DEFAULT
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 19
ms for event 94f2a7fe_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 1 ms
for event 94f2a7fe_TASK
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 7a2b6143_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 7a2b6143_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7a2b6143_DEFAULT
Start computing new ideal states for resources: []
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
54 ms for event 94f2a7fe_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 94f2a7fe_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 94f2a7fe_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 94f2a7fe_TASK
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 94f2a7fe_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 94f2a7fe_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END: Invoking TASK controller pipeline for event MessageChange::94f2a7fe_TASK for
cluster PinotCluster, took 170 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 86c7f301_TASK. Event session ID: 3000a812cfc0036
Event 86c7f301_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 86c7f301_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 86c7f301_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 86c7f301_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::86c7f301_TASK : Refreshed 9 property StateModelDefinition
took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::86c7f301_TASK : Refreshed 1 property ClusterConstraint
took 1 ms. Selective: false
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 22 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
23 ms for event 7a2b6143_DEFAULT
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 32
ms for event 7a2b6143_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 7a2b6143_DEFAULT
No throttle config is set!
Event PinotCluster::TASK::86c7f301_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::86c7f301_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110375121, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
1 ms for event 7a2b6143_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 7a2b6143_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7a2b6143_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 7a2b6143_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 7a2b6143_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 7a2b6143_DEFAULT
END: Invoking DEFAULT controller pipeline for event
IdealStateChange::7a2b6143_DEFAULT for cluster PinotCluster, took 234 ms
Callback time for event: IdealStateChange took: 0 ms
InQueue time for event: IdealStateChange took: 417 ms
TotalProcessed time for event: IdealStateChange took: 234 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 29b15e3b_DEFAULT. Event session ID: 3000a812cfc0036
Event 29b15e3b_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 29b15e3b_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 29b15e3b_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 29b15e3b_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : Refreshed 9 property
StateModelDefinition took 3 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : Refreshed 1 property
ClusterConstraint took 1 ms. Selective: false
START: InstanceMessagesCache.refresh()
Event PinotCluster::TASK::86c7f301_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::86c7f301_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110375145, took 20 ms
Event 86c7f301_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110375096 took 70 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 70 ms
for event 86c7f301_TASK
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 37
ms.
Start to refresh stale message cache
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 37
ms for event 86c7f301_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 86c7f301_TASK
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110375199, took 23 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 29b15e3b_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::29b15e3b_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110375223, took 0 ms
Event 29b15e3b_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110375156 took 73 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 73 ms
for event 29b15e3b_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
19 ms for event 29b15e3b_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29b15e3b_DEFAULT
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
52 ms for event 86c7f301_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 1 ms for
event 86c7f301_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 86c7f301_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 86c7f301_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 86c7f301_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 86c7f301_TASK
END: Invoking TASK controller pipeline for event CurrentStateChange::86c7f301_TASK
for cluster PinotCluster, took 160 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking TASK controller pipeline for cluster: PinotCluster. Event type:
MessageChange, ID: 29b15e3b_TASK. Event session ID: 3000a812cfc0036
Event 29b15e3b_TASK : No ideal state change for PinotCluster cluster, TASK pipeline
Event 29b15e3b_TASK : No live instance change for PinotCluster cluster, TASK
pipeline
Event 29b15e3b_TASK : No instance config change for PinotCluster cluster, TASK
pipeline
Event 29b15e3b_TASK : No resource config change for PinotCluster cluster, TASK
pipeline
Event PinotCluster::TASK::29b15e3b_TASK : Refreshed 9 property StateModelDefinition
took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::TASK::29b15e3b_TASK : Refreshed 1 property ClusterConstraint
took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 20
ms.
Start to refresh stale message cache
Event PinotCluster::TASK::29b15e3b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::29b15e3b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110375281, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 60 ms for event 29b15e3b_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 29b15e3b_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 29b15e3b_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 20 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 29b15e3b_DEFAULT
Event PinotCluster::TASK::29b15e3b_TASK : 0 properties refreshed from ZK.
Event PinotCluster::TASK::29b15e3b_TASK : END: participantStateCache.refresh() for
cluster PinotCluster, started at : 1730110375305, took 31 ms
Event 29b15e3b_TASK : END: WorkflowControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110375256 took 81 for TASK pipeline
END ReadClusterDataStage for TASK pipeline for cluster PinotCluster. took: 81 ms
for event 29b15e3b_TASK
END ResourceComputationStage for TASK pipeline for cluster PinotCluster. took: 18
ms for event 29b15e3b_TASK
END ResourceValidationStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29b15e3b_TASK
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 29b15e3b_DEFAULT
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 11
ms for event 29b15e3b_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 29b15e3b_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 29b15e3b_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29b15e3b_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 29b15e3b_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 29b15e3b_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 29b15e3b_DEFAULT
END: Invoking DEFAULT controller pipeline for event MessageChange::29b15e3b_DEFAULT
for cluster PinotCluster, took 228 ms
Callback time for event: MessageChange took: 1 ms
InQueue time for event: MessageChange took: 185 ms
TotalProcessed time for event: MessageChange took: 228 ms
Controller stopping periodical rebalance timer at period 9223372036854775807
START: Invoking DEFAULT controller pipeline for cluster: PinotCluster. Event type:
CurrentStateChange, ID: 86c7f301_DEFAULT. Event session ID: 3000a812cfc0036
Event 86c7f301_DEFAULT : No ideal state change for PinotCluster cluster, DEFAULT
pipeline
Event 86c7f301_DEFAULT : No live instance change for PinotCluster cluster, DEFAULT
pipeline
Event 86c7f301_DEFAULT : No instance config change for PinotCluster cluster,
DEFAULT pipeline
Event 86c7f301_DEFAULT : No resource config change for PinotCluster cluster,
DEFAULT pipeline
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : Refreshed 9 property
StateModelDefinition took 2 ms. Selective: false
overwrite existing constraint-value. old-value: null, new-value: 100000
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : Refreshed 1 property
ClusterConstraint took 2 ms. Selective: false
START: InstanceMessagesCache.refresh()
END CurrentStateComputationStage for TASK pipeline for cluster PinotCluster. took:
53 ms for event 29b15e3b_TASK
AssignableInstanceManager built AssignableInstances from scratch based on
CurrentState.
Current quota capacity: {"Server_pinot-server-12.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-11.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-10.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-9.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-13.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-15.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-14.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.226.104_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.228.237_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-16.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-2.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.195_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-0.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.225.15_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Minion_10.5.234.190_9514":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Broker_10.5.234.128_8099":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-0.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-2.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-3.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-1.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-8.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-7.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-4.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-6.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":
{"DEFAULT":"0/40"}},"Server_pinot-server-5.pinot-
server.pinot.svc.cluster.local_8098":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}}
END TaskSchedulingStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 29b15e3b_TASK
START TaskPersistDataStage.process()
END TaskPersistDataStage.process() for cluster PinotCluster took 0 ms
END TaskPersistDataStage for TASK pipeline for cluster PinotCluster. took: 0 ms for
event 29b15e3b_TASK
Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker
END TaskGarbageCollectionStage for TASK pipeline for cluster PinotCluster. took: 0
ms for event 29b15e3b_TASK
START AsyncProcess: TASK::TaskGarbageCollectionStage
END MessageGenerationPhase for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29b15e3b_TASK
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
END TaskMessageDispatchStage for TASK pipeline for cluster PinotCluster. took: 0 ms
for event 29b15e3b_TASK
END: Invoking TASK controller pipeline for event MessageChange::29b15e3b_TASK for
cluster PinotCluster, took 152 ms
END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 19
ms.
Start to refresh stale message cache
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110375408, took 24 ms
END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms.
Event 86c7f301_DEFAULT : No customized state config change for PinotCluster
cluster, DEFAULT pipeline
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : 0 properties refreshed from ZK.
Event PinotCluster::DEFAULT::86c7f301_DEFAULT : END:
participantStateCache.refresh() for cluster PinotCluster, started at :
1730110375432, took 0 ms
Event 86c7f301_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster
PinotCluster, started at 1730110375384 took 55 for DEFAULT pipeline
END ReadClusterDataStage for DEFAULT pipeline for cluster PinotCluster. took: 55 ms
for event 86c7f301_DEFAULT
END ResourceComputationStage for DEFAULT pipeline for cluster PinotCluster. took:
20 ms for event 86c7f301_DEFAULT
END ResourceValidationStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86c7f301_DEFAULT
END CurrentStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 55 ms for event 86c7f301_DEFAULT
END CustomizedStateComputationStage for DEFAULT pipeline for cluster PinotCluster.
took: 0 ms for event 86c7f301_DEFAULT
Submitted asynchronous DEFAULT::TopStateHandoffReportStage task to worker
END TopStateHandoffReportStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 86c7f301_DEFAULT
START AsyncProcess: DEFAULT::TopStateHandoffReportStage
Submitted asynchronous DEFAULT::ExternalViewComputeStage task to worker
END ExternalViewComputeStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86c7f301_DEFAULT
START AsyncProcess: DEFAULT::ExternalViewComputeStage
Start computing new ideal states for resources: []
Start emergency rebalance.
Finish emergency rebalance
Finish computing new ideal states for resources: []
Start calculating the new best possible assignment.
Start calculating for an assignment with algorithm ConstraintBasedAlgorithm
Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0
ms.
Finish calculating the new best possible assignment.
END AsyncProcess: DEFAULT::TopStateHandoffReportStage, took 21 ms
END BestPossibleStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
24 ms for event 86c7f301_DEFAULT
END MessageGenerationPhase for DEFAULT pipeline for cluster PinotCluster. took: 30
ms for event 86c7f301_DEFAULT
END AsyncProcess: DEFAULT::ExternalViewComputeStage, took 59 ms
END MessageSelectionStage for DEFAULT pipeline for cluster PinotCluster. took: 12
ms for event 86c7f301_DEFAULT
No throttle config is set!
END IntermediateStateCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 86c7f301_DEFAULT
END MessageThrottleStage for DEFAULT pipeline for cluster PinotCluster. took: 6 ms
for event 86c7f301_DEFAULT
Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker
END MaintenanceRecoveryStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86c7f301_DEFAULT
START AsyncProcess: DEFAULT::MaintenanceRecoveryStage
END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms
END ResourceMessageDispatchStage for DEFAULT pipeline for cluster PinotCluster.
took: 5 ms for event 86c7f301_DEFAULT
Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker
END PersistAssignmentStage for DEFAULT pipeline for cluster PinotCluster. took: 0
ms for event 86c7f301_DEFAULT
START AsyncProcess: DEFAULT::PersistAssignmentStage
START AsyncProcess: DEFAULT::TargetExteralViewCalcStage
END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms
Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker
END TargetExteralViewCalcStage for DEFAULT pipeline for cluster PinotCluster. took:
0 ms for event 86c7f301_DEFAULT
END AsyncProcess: DEFAULT::PersistAssignmentStage, took 0 ms
END: Invoking DEFAULT controller pipeline for event
CurrentStateChange::86c7f301_DEFAULT for cluster PinotCluster, took 207 ms
Callback time for event: CurrentStateChange took: 0 ms
InQueue time for event: CurrentStateChange took: 417 ms
TotalProcessed time for event: CurrentStateChange took: 207 ms
Starting TaskMetricsEmitter with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: TaskMetricsEmitter
[TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 0ms
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Starting RebalanceChecker with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: RebalanceChecker
Processing 3 tables in task: RebalanceChecker
Finish processing 0/3 tables in task: RebalanceChecker
[TaskRequestId: auto] Finish running task: RebalanceChecker in 2ms
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables/amxordersgpx/schema,
content-type null status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tenants, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/cluster/configs, content-type
null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/brokers/tenants/
DefaultTenant, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tenants/AMX?type=server,
content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tenants/CLICKSTREAM/tables,
content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/
Broker_10.5.234.128_8099, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-15.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-14.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-16.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-3.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-6.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/
Minion_10.5.234.195_9514, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables/clientlogin_REALTIME/
idealstate, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: clientlogin_REALTIME with timeout:
30000ms
Finished reading information for table: clientlogin_REALTIME
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables/clientlogin_REALTIME/
size, content-type null status code 200 OK
Reading segment sizes from 4 servers for table: positionamxntt_REALTIME with
timeout: 30000ms
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables/
positionamxntt_REALTIME/externalview, content-type null status code 200 OK
Finished reading information for table: positionamxntt_REALTIME
Handled request from 10.5.229.104 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables/
positionamxntt_REALTIME/size, content-type null status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Scan statusUpdates and errors for cluster: PinotCluster, by controller:
org.apache.helix.manager.zk.ZKHelixManager@4e4f4092
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Starting SegmentStatusChecker with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: SegmentStatusChecker
Processing 3 tables in task: SegmentStatusChecker
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
 metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624411
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2--2147483648
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624468
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-
info,id=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-data-v2
at
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithR
epository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam
icMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec
t(DefaultMBeanServerInterceptor.java:895) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean
(DefaultMBeanServerInterceptor.java:320) ~[?:?]
at
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer
.java:523) ~[?:?]
at
org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo
(AppInfoParser.java:64) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:814) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:665) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:646) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:626) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createC
onsumer(KafkaPartitionLevelConnectionHandler.java:83) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(
KafkaPartitionLevelConnectionHandler.java:70) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStre
amMetadataProvider.java:59) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadata
Provider(KafkaConsumerFactory.java:38) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(St
reamMetadataProvider.java:91) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:70) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:31) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGrou
pMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.realtime.MissingConsumingSegmentFinder.<init
>(MissingConsumingSegmentFinder.java:79) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.updateSegmentMetrics(Segment
StatusChecker.java:375) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:124) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:66) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processT
ables(ControllerPeriodicTask.java:116) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(
ControllerPeriodicTask.java:79) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTas
kScheduler.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
ScheduledThreadPoolExecutor.java:305) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1136) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-0
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624576
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-1
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624687
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-2
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624798
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-3
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110624907
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-4
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625016
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-5
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625129
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-6
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625241
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-7
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [amx-cnf-kafka-broker2.ntt.prod.angelone.in:9094, amx-
cnf-kafka-broker1.ntt.prod.angelone.in:9094, amx-cnf-kafka-
broker0.ntt.prod.angelone.in:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625352
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Subscribed to partition(s): amx-order-data-v2-8
[Consumer clientId=PartitionGroupMetadataFetcher-amxordersntt_REALTIME-amx-order-
data-v2, groupId=null] Cluster ID: r6ejb632T2O-TE80nW2iAA
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-amxordersntt_REALTIME-
amx-order-data-v2 unregistered
Reading segment sizes from 6 servers for table: amxordersntt_REALTIME with timeout:
30000ms
Finished reading information for table: amxordersntt_REALTIME
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
 fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
 ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625924
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios--2147483648
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
 internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625928
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-
info,id=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
at
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithR
epository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam
icMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec
t(DefaultMBeanServerInterceptor.java:895) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean
(DefaultMBeanServerInterceptor.java:320) ~[?:?]
at
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer
.java:523) ~[?:?]
at
org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo
(AppInfoParser.java:64) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:814) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:665) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:646) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:626) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createC
onsumer(KafkaPartitionLevelConnectionHandler.java:83) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(
KafkaPartitionLevelConnectionHandler.java:70) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStre
amMetadataProvider.java:59) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadata
Provider(KafkaConsumerFactory.java:38) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(St
reamMetadataProvider.java:91) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:70) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:31) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGrou
pMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.realtime.MissingConsumingSegmentFinder.<init
>(MissingConsumingSegmentFinder.java:79) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.updateSegmentMetrics(Segment
StatusChecker.java:375) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:124) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:66) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processT
ables(ControllerPeriodicTask.java:116) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(
ControllerPeriodicTask.java:79) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTas
kScheduler.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
ScheduledThreadPoolExecutor.java:305) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1136) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-0
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625935
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-1
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625941
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-2
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625951
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-3
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625959
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-4
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625966
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-5
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625973
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-6
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625979
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-7
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625987
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-8
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110625993
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-9
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626001
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-10
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626007
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-11
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626015
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-12
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626022
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-13
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626028
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-14
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626036
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-15
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626043
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-16
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626051
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-17
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626059
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-18
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626067
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-19
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626075
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-20
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626082
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-21
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626090
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-22
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
 enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
 ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626097
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Subscribed to partition(s): spark_ios-23
[Consumer clientId=PartitionGroupMetadataFetcher-clickstream_REALTIME-spark_ios,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clickstream_REALTIME-
spark_ios unregistered
Reading segment sizes from 7 servers for table: clickstream_REALTIME with timeout:
30000ms
Finished reading information for table: clickstream_REALTIME
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
 ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626127
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login--2147483648
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
 client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
 ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626132
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-
info,id=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
at
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithR
epository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam
icMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec
t(DefaultMBeanServerInterceptor.java:895) ~[?:?]
at
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean
(DefaultMBeanServerInterceptor.java:320) ~[?:?]
at
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer
.java:523) ~[?:?]
at
org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo
(AppInfoParser.java:64) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:814) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:665) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:646) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(Kafk
aConsumer.java:626) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createC
onsumer(KafkaPartitionLevelConnectionHandler.java:83) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(
KafkaPartitionLevelConnectionHandler.java:70) ~[pinot-kafka-2.0-1.2.0-
shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStre
amMetadataProvider.java:59) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadata
Provider(KafkaConsumerFactory.java:38) ~[pinot-kafka-2.0-1.2.0-shaded.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(St
reamMetadataProvider.java:91) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:70) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetada
taFetcher.java:31) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGrou
pMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.realtime.MissingConsumingSegmentFinder.<init
>(MissingConsumingSegmentFinder.java:79) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
 at
org.apache.pinot.controller.helix.SegmentStatusChecker.updateSegmentMetrics(Segment
StatusChecker.java:375) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:124) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusCh
ecker.java:66) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processT
ables(ControllerPeriodicTask.java:116) ~[pinot-all-1.2.0-jar-with-
dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(
ControllerPeriodicTask.java:79) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTas
kScheduler.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
ScheduledThreadPoolExecutor.java:305) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1136) [?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-0
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
 allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
 session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626138
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-1
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
 allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
 session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626146
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-2
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
 allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
 session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626154
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-3
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
 allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
 session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626160
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-4
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Skipping auto SSL server validation since it's not configured.
ConsumerConfig values:
 allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [b-1.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-2.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092, b-3.kafkacnfprod2.sbtdrb.c2.kafka.ap-south-
1.amazonaws.com:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.pinot.shaded.org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
 session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.pinot.shaded.org.apache.kafka.common.serialization.BytesDeserializer

The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a

known config.
The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a
known config.
The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known
config.
The configuration 'streamType' was supplied but isn't a known config.
The configuration 'stream.kafka.consumer.prop.request.timeout.ms' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.prop.auto.commit.enable' was supplied but
isn't a known config.
The configuration 'stream.kafka.consumer.type' was supplied but isn't a known
config.
The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a
known config.
The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but
isn't a known config.
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1730110626167
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Subscribed to partition(s): client-login-5
[Consumer clientId=PartitionGroupMetadataFetcher-clientlogin_REALTIME-client-login,
groupId=null] Cluster ID: Bj1atX7tR-WJwTk-7ArZPQ
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Metrics scheduler closed
Closing reporter
org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-clientlogin_REALTIME-
client-login unregistered
Reading segment sizes from 1 servers for table: clientlogin_REALTIME with timeout:
30000ms
Finished reading information for table: clientlogin_REALTIME
Finish processing 3/3 tables in task: SegmentStatusChecker
[TaskRequestId: auto] Finish running task: SegmentStatusChecker in 3698ms
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Starting TaskMetricsEmitter with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: TaskMetricsEmitter
[TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 0ms
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Starting RebalanceChecker with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: RebalanceChecker
Processing 3 tables in task: RebalanceChecker
Finish processing 0/3 tables in task: RebalanceChecker
[TaskRequestId: auto] Finish running task: RebalanceChecker in 1ms
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Session 0x2000a8125260033 for server zookeeper/172.20.157.33:2181, Closing socket
connection. Attempting reconnect except it is a SessionExpiredException.
java.io.IOException: Packet len 1637600 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:124)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:84)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274)
[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-
cc33ac502a02e2fe830fe21e556234ee99351a7a]
zkclient 9, zookeeper state changed ( Disconnected )
KeeperState:Disconnected, SessionId: 2000a8125260033, instance: Controller_pinot-
controller-1.pinot-controller.pinot.svc.cluster.local_9000, type: PARTICIPANT
zkClient to zookeeper:2181 is not connected, wait for 10000ms.
Opening socket connection to server zookeeper/172.20.157.33:2181.
SASL config status: Will not attempt to authenticate using SASL (unknown error)
Socket connection established, initiating session, client: /10.5.227.76:56500,
server: zookeeper/172.20.157.33:2181
Session establishment complete on server zookeeper/172.20.157.33:2181, session id =
0x2000a8125260033, negotiated timeout = 30000
zkclient 9, zookeeper state changed ( SyncConnected )
KeeperState: SyncConnected, instance: Controller_pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, type: PARTICIPANT
Getting Helix leader: pinot-controller-1.pinot-
controller.pinot.svc.cluster.local_9000, Helix version: 1.3.1, mtime: 1729238439629
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tables?type=realtime,
content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/brokers/tenants/
DefaultTenant, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tenants/DefaultTenant/tables,
content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/tenants/AMX/tables, content-
type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Controller_pinot-
controller-0.pinot-controller.pinot.svc.cluster.local_9000, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/
Broker_10.5.234.128_8099, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/
Broker_10.5.228.237_8099, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-13.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-10.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-16.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-8.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Server_pinot-
server-7.pinot-server.pinot.svc.cluster.local_8098, content-type null status code
200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/zk/ls?path=%2FPinotCluster
%2FLIVEINSTANCES, content-type null status code 200 OK
Handled request from 10.5.225.229 GET http://internal-k8s-pinot-pinotcon-
1114a22f57-1757926636.ap-south-1.elb.amazonaws.com:80/instances/Controller_pinot-
controller-2.pinot-controller.pinot.svc.cluster.local_9000, content-type null
status code 200 OK
Handled request from 10.5.225.229 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.232.26 GET http://10.5.227.76:9000/, content-type null
status code 200 OK
Handled request from 10.5.229.104 GET http://10.5.227.76:9000/, content-type null
status code 200 OK