Decide on must-gather.sh inclusion in Dockerfile after distroless migration

[rajathagasthya]

Part of NVIDIA/cloud-native-team#299. Deferred from PR #2434.

hack/must-gather.sh is a 286-line bash script copied into the image
at docker/Dockerfile:103 as /usr/bin/gather, the OpenShift
oc adm must-gather --image=… plugin entrypoint. It shells out to
kubectl/oc, neither of which ship in the distroless base today.
The script is therefore largely unusable from inside the pod already,
and once the image base drops -dev (no shell), it cannot run at all.

Decide and implement one of:

1. Rewrite the script in Go as a subcommand of nvidia-validator
   or a new dedicated binary. Use a vendored Go Kubernetes client.
   Stays inside the gpu-operator distroless image.
2. Split must-gather into its own image — a small RHEL/UBI-based
   image that vendors oc and the bash script. Document a separate
   image tag for the must-gather plugin.
3. Remove /usr/bin/gather from the gpu-operator image entirely
   and document that customers should run must-gather.sh directly
   from outside the cluster. Customers report this is what they
   already do in practice.

Acceptance:

• Decision is made and recorded
• Customer-facing must-gather workflow is documented

[tariq1890]

My preference is to keep must-gather.sh as a shell script. We don't really need to include it in our application containers, right?

[rajathagasthya]

We don't really need to include it in our application containers, right?

Correct. This would make things simple. I wasn't sure why it's included in the container image currently.

[cdesiniotis]

I don't recall the original reasoning for including this script in the image. I would also vote to remove it.

[rajathagasthya]

Decision: option 3 — remove /usr/bin/gather from the gpu-operator image entirely. Customers already run hack/must-gather.sh from outside the cluster against an existing kubeconfig; the in-image entrypoint depended on bash, kubectl, and oc (none of which ship in the distroless base) and was already largely unusable.

The script itself stays in the repo for customer use. The COPY hack/must-gather.sh /usr/bin/gather line has been dropped in PR #2434, which now also resolves this issue.

[rajathagasthya]

Closing — under the new direction we keep the shell in the image, so must-gather.sh stays in /usr/bin/gather. No removal needed. See NVIDIA/cloud-native-team#299 for the updated approach.