Add gpu metrics to TraceRecord

[FloWuenne]

Description

This PR implements feature requests from #4286 .

The current implementation adds the following fields to the TraceRecord:

• gpu_model
• gpu_mem
• gpu_driver: 'str'
• -'%gpu'
• %gpu_mem
• avg_gpu_mem

This is accomplished by implementing a new nxf_gpu_watch() in command-trace.txt , similar to nxf_mem_watch(). This allows to access GPU metrics in the Tracefile, by adding the following lines to the nextflow.config:

trace{
fields = 'gpu_model,gpu_mem,gpu_driver,%gpu,%gpu_mem,avg_gpu_mem'
}

Limitations

A current limitation of this approach is that if using Nextflow with an executor (for example local) and a local GPU that is operating system wide, the usage metrics are not task specific but include background processes. For example, on my local Ubuntu machine with an NVIDIA GeForce GTX 1650 even a cpu only task reported some GPU usage, due to background GPU processes utilizing the GPU. This should not be the case when running on instances where GPUs are assigned to specific tasks using accelerator, however I haven't tested this on multi-GPU machines on AWS or other cloud executors.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	530d9a4
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67eae7ba6c86b00008b2df36
😎 Deploy Preview	https://deploy-preview-5930--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pditommaso]

This looks a nice step forward, I'm start wondering if it's not time we write our own tool to trace these metrics (including gpu). @jordeu @fntlnz thoughts?

[jordeu]

This looks a nice step forward, I'm start wondering if it's not time we write our own tool to trace these metrics (including gpu). @jordeu @fntlnz thoughts?

Absolutely. Our own tool will simplify Nextflow bash scripts and allow us to do many more things, like collecting metrics during the process's lifetime.

[FloWuenne]

Since our own tool for performance metrics still seems a bit away though, do we want to move forward in the meantime with using nvidia-smi here and add more gpu controllers over time or are there major implementations here we want to change?

[fntlnz]

This looks a nice step forward, I'm start wondering if it's not time we write our own tool to trace these metrics (including gpu). @jordeu @fntlnz thoughts?

Agreed, seeing the broader picture with the ability to drill down to the specific task would be very useful both to optimize pipelines and also to optimize their runtime scheduling.

[pditommaso]

@FloWuenne this is a nice POC, but I fear the shell script is becoming to big/complex. Also I would like to avoid introducing another background process to monitor GPU stats. I talked with @jordeu are we are going to implement a new tracing tool binary compiled taking inspiration from this PR.

[FloWuenne]

Sounds good @pditommaso! Looking forward to see the new tracing tool soon!