Better observability for subscriptions

[lutter]

We need to gather better metrics for the steps we take from updating a subgraph to sending out an update for any subscriptions for that subgraph. For that, we should

• Add a timestamp to StoreEvent that gets set when the event is created
• Add Prometheus metrics that measure time since event creation at the following points when handling StoreEvents (ideally per subgraph):
  • StoreEvent is sent (i.e., NOTIFY is called)
  • StoreEvent is received in store::postgres::NotificationListener
  • StoreEvent is received in the GraphQL layer in graphql::subscription::execute_subscription_event
  • Subscription update is sent in server::websocket::connection::handle_incoming_message by putting it into the result_sink This last step will require that the timestamp of the StoreEvent is returned in the response stream together with the actual query result

For each step in the process we'd want to define how much processing time is acceptable, and bin Prometheus values accordingly. Some work needs to go into understanding what realistic values for these are.

[lutter]

Subscriptions are no longer supported