grpc/sa: Implement deep health checks

[beautifulentropy]

Add the necessary scaffolding for deep health checking of our various gRPC components. Each component implementation that also implements the grpc.checker interface will be checked periodically, and the health status of the component will be updated accordingly.

Add the necessary methods to SA to implement the grpc.checker interface and register these new health checks with Consul.

Additionally:

• Update entry point script to check for ProxySQL readiness.
• Increase the poll rate for gRPC Consul checks from 5s to 2s to help with DNS failures, due to check failures, on startup.
• Change log level for Consul from INFO to ERROR to deal with noisy logs full of transport failures due to Consul gRPC checks firing before the SAs are up.

Fixes #6878
Part of #6795

I thought I understood this, but I don't yet.

[beautifulentropy]

@pgporada and I had a call to chat through his questions.

[beautifulentropy]

Took some screenshots and ran through a few scenarios so folks are clear on how this is going to work:

• grpc will always be green if the grpc server is up
• sa_ro failure is a failure for sa and sa_ro
• sa failure is just a failure for sa

The DB and ProxySQL are both up, everything is green:

If you take down ProxySQL (or the DB), sa and saro are red, but grpc is green:

If you remove the sa_ro user from ProxySQL, sa and saro are red, but grpc is green:

If you remove the sa user from ProxySQL, sa is red, but saro and grpc are green:

[aarongable]

Mostly tiny code organization comments. Last top-level comment: it would be great to have a unittest that tests the check closure: sets up a health server, supplies a tiny fake health check function, and ensures that the health server's status was updated appropriately. This may require busting check all the way out to being a method on serverBuilder or something like that, rather than being a closure =/