Simulated failures (aka chaos testing) in KIC tests

[mflendrich]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

Our e2e tests should try to mimic a production lifecycle of components within the cluster, the controller, and the proxy. We should introduce chaos (chaos testing) to simulate production cluster problems.

Proposed Solution

• tests of our APIs in the e2e environment should be pointed at real applications (e.g. not httpbin) doing validated work
• the cluster should have its resources exhausted at various points, semi-arbitrarily
• the cluster network should be saturated at various points
• nodes should be randomly killed without warning during the tests

Additional information

Broken out of #1605 originally reported by @shaneutt.
This issue is a rewrite that narrows down the scope and indicates the experimental nature of this enhancement.

Acceptance Criteria

• A suite of chaos-induced tests is defined, possibly inspired by/extending the above, and documented on this issue or in a KEP/design doc.
• Issues for individual planned test cases are created.