[improvement] : Capture nvidiadriver CRs as well

[rahulait]

Changes to must-gather script include:

1. Additional step to capture nvidiadriver CRs as well
2. Capture pod logs with timestamps. Without this, for ex for driver pods, its difficult to tell when the pod was restarted and when those log lines were written.
3. Add labels to filter driver pods in case driver daemonsets are provisioned using nvidiadriver. This will now run nvidia-bug-report script for nodes where driver pod is provisioned by nvidiadriver CR
4. Add logic to capture which processes are using GPU if any. This will help in identifying cases when we don't expect anyone to be using GPU

Description

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

[rahulait]

@cdesiniotis @rajathagasthya re-requesting review as I added one more commit which now adds labelselector for nvidiadriver pods so that we capture nvidia-bug-report output from nodes as well when nvidiadriver cr is used. I also added some additional logic to capture GPU usage on nodes for easier troubleshooting of scenarios where someone outside of gpu-operator took access to GPU and causes issues with driver install/upgrades.

It can be tested using:

curl -fsSL https://raw.githubusercontent.com/rahulait/gpu-operator/refs/heads/update-must-gather/hack/must-gather.sh | bash

[tariq1890]

Can you amend the commit message to "[improvement] capture NVIDIADriver CRs in must-gather" ?

[rahulait]

Can you amend the commit message to "[improvement] capture NVIDIADriver CRs in must-gather" ?

Yup, updated.

[karthikvetrivel]

Thanks for taking a look at this, Rahul. Looks great.