feat(meshservice): mitigate and handle resource conflicts

[jakubdyszkiewicz]

Checklist prior to review

Fix #10511

I think the best strategy for now is to

• Introduce jitter so not all components start at once
• Log on info, not on ERROR

Lock might be too aggressive and batching updates are complicated to implement for now.
I noticed that in E2E jitter already solves a lot of such issues. We can revisit other strategies if needed

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Don't forget ci/ labels to run additional/fewer tests
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --

[Icarus9913]

Further thought:
Maybe we could also use the kubernetes Retry function to handle with our update operation in the future?

https://github.com/kubernetes/apimachinery/blob/4e174c5e3655ac47294239904b796671c5d9dfd6/pkg/util/wait/backoff.go#L459

[jakubdyszkiewicz]

@Icarus9913 we already have .Upsert() function with retry backoff and jitter. The problem is that you need to first retrieve the resource on retry so writing this logic is a bit meh when you don't want to pay the cost upfront for retrieving a resource because you already fetched the list