spawn a new deis cluster (in a new namespace) for each run

[arschles]

this one may be involved, but it would be nice to create a new deis cluster in its own namespace for each test run, and then tear it down at the end. this would eliminate a whole class of state management issues across test suite runs, and also allow us to run multiple suite concurrently on the same k8s cluster.

also note that I believe we have some outstanding issues that prevent us from running 2+ deis clusters reliably on the same k8s cluster, so those will have to be cleared before this can be possible.

Purposefully not assigned a milestone, as this is a nice to have and not critical.

[technosophos]

Can we run on helm HEAD and see if the new template logic will make this easy? There are random functions available to templates that might give us exactly this feature.

[arschles]

@technosophos yea, that'd work. we can have the BeforeSuite issue the helm install with the proper template values, and then wait until all pods are up and running before continuing

[mboersma]

I think provisioning should remain separate from the tests themselves this time. One reason is that the e2e tests had also been envisioned as a "sanity check" on a user's newly installed cluster before it's put into service, and in that case automatically provisioning / decommissioning would interfere. Another use case that past experience shows is important was captured by the SKIP_CLEANUP env var in the "old" integration tests: provision and run the tests but leave everything intact for further forensic examination.

I don't feel that strongly about this, but I can see value to keeping provisioning and testing separate. Or at the least, allowing the helm parts of BeforeSuite/AfterSuite to be skipped if we do follow through with this proposal.

[arschles]

One reason is that the e2e tests had also been envisioned as a "sanity check" on a user's newly installed cluster before it's put into service, and in that case automatically provisioning / decommissioning would interfere

@mboersma excellent point. 👍

Or at the least, allowing the helm parts of BeforeSuite/AfterSuite to be skipped if we do follow through with this proposal.

I'm ok with either way. Leaning toward putting the helm install + wait for pods logic into a completely separate program because I imagine it could be reusable down the line. Also it's more unixey.

Regardless, I do think we should follow through with this proposal in the beta cycle. Would be nice to see these tests run as part of our pipeline at some point.

[jchauncey]

See #142