CI: More useful checks?

[sadlerap]

Our current CI testing for conformance testing isn't going to work long-term in my opinion. Running against an empty cluster isn't going to yield meaningful results: while it does run the code, execution stops as soon as it actually tries to resolve a service binding since there isn't anything doing the resolution.

As it stands, I think we have a few options for replacements:

• Remove this aspect of the CI or make it non-required for pull requests. (Use the reference implementation to test conformance #12)
• Begin writing unit tests for our testing code.
• Implement linting & static analysis checks in place to help ensure that the tests don't rot.

I'm not super happy about the second option, since testing tests doesn't seem like a good path to me. The other options feel like good places for improvement.

Thoughts?

[scothis]

Can we run the conformance tests against a pinned "latest" release of the reference implementation?

Any particular implementation will likely have most test pass (hopefully), but there may also be tests that fail. This is ok as long as we can capture that a particular test is expected to fail. Passing this repo's CI means that the conformance test that are expected to pass, pass; and the test that are expected to fail, fail.

[sadlerap]

When we make a release of the reference implementation, this should be feasible. We should leave it to the implementations that we test to declare which tests they expect to pass and fail.

[baijum]

which tests they expect to pass and fail.

For implementations to selectively run conformance tests, we can add different classifications in the form of Behave tags. Few examples:

• @core
• @direct-secret-reference
• @workload-resource-mapping
• @rbac
• @webhook
• @immutable-resource
• @label-selector
• @kube.version.1.21
• @kube.version.1.22