I think would be useful to display following additional metrics especially in terminal servers or transcoding servers.
in the top, system informations:
1. add a circle for each GPU installed in the system plotting current GPU load percentage
2. one row CPU Fan speed
3. one row GPU information with device info
4. one row GPU current temperature
5. one row GPU Fan speed
and below in Stats History:
6. GPUs load percentage (one graph for each present integrated/discrete
7. GPUs VRAM usage (one graph for each present integrated/discrete
I was thinking nvtop would be the easiest tool to get those info
GPU is so inconsistent, it’s hard to support. They all have different field names, often aren’t supported by sensors, or aren’t supported without running sensors-detect (which can, in some rare circumstances, crash the system or one or more devices that it probes) and adding the specific modules you need.
Not saying we can’t/won’t make an attempt, but also trying to manage expectations. It’s complicated.
And, a caveat: If you’re not able to get the info you want from sensors, we won’t be able to get it either. Not every GPU provides reasonable data in sensors, AFAIK, even with the drivers loaded.
And, if there isn’t a tool that already provides VRAM data (e.g. rocm-smi for AMD devices), we’re not going to try to spelunk in the low-level drivers to figure out how to get that data.
If you want to see GPU data for a particular GPU or family of GPUs, you should share how you get the data on the command line and what its output looks like. We don’t own all the GPUs. I can only test against AMD stuff, as that’s what I have here.
Oh, actually, it looks like everyone except nvidia with their proprietary drivers does provide data via the usual Linux channels (under the /sys filesystem). So, I think it’s mainly only a PITA for nvidia if you use the proprietary drivers (the open source Nouveau drivers participate in the Linux monitoring features correctly, AFAIK).
Can y’all run this script on the servers you want to monitor and see if what it returns is correct? Note GPU fan speed is rarely available (if ever). I expect N/A for most devices, I think. But, if you can find a way to get GPU fan speed for the various GPUs, I’m all ears.