CPU Running Near 100%

Over the past few days, my server is running at close to 100%.

In the past week, there have been some updates and I have added a couple domains. One of them is primarily for email, so when I saw clamav running wild, I stopped that and maildet.

Now, it is still running high and the biggest culprit is:
107 % | /usr/share/usermin/authentic-theme/session_login.cgi

Top -i shows:

top - 15:08:01 up  2:15,  0 users,  load average: 34.55, 44.00, 43.74
Tasks: 522 total,  29 running, 492 sleeping,   0 stopped,   1 zombie
%Cpu(s): 56.2 us, 34.5 sy,  0.0 ni,  6.8 id,  0.5 wa,  0.0 hi,  2.1 si,  0.0 st
MiB Mem :  48171.3 total,  18989.9 free,   9777.1 used,  19404.3 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  36286.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                               
      1 root      20   0  168712  14272   8356 S  94.9   0.0  73:40.83 systemd                                                                                                                                                               
 136961 root      20   0   66784  58220   8064 R  70.1   0.1   0:02.23 /usr/share/webm                                                                                                                                                       
   1608 mysql     20   0   10.6g   1.7g  27024 S  57.9   3.6  79:15.69 mariadbd                                                                                                                                                              
 136974 root      20   0   67340  58996   8120 R  47.6   0.1   0:01.48 /usr/share/webm                                                                                                                                                       
 135746 naf       20   0  749608 220672 135112 R  38.9   0.4   0:05.11 php-fpm8.4                                                                                                                                                            
  50565 lenovor+  20   0  764608 248320 215064 R  38.3   0.5   4:11.82 php-fpm8.3                                                                                                                                                            
 136984 root      20   0   57364  48696   8004 R  34.4   0.1   0:01.07 /usr/share/webm                                                                                                                                                       
 131367 gis-uk    20   0  711580 168856 135104 R  34.1   0.3   0:22.69 php-fpm8.4                                                                                                                                                            
 131182 abigudac  20   0  708820  70892  24784 R  32.8   0.1   0:28.89 php-fpm8.4                                                                                                                                                            
  75121 borough+  20   0  694336 100992  69944 R  31.2   0.2   0:28.44 php-fpm8.4                                                                                                                                                            
 136926 root      20   0       0      0      0 Z  30.2   0.0   0:02.40 /usr/share/webm                                                                                                                                                       
 135179 antique+  20   0  771976 241228 132396 R  29.6   0.5   0:20.22 php-fpm8.4                                                                                                                                                            
  59998 guide     20   0  674564 107844  90248 R  29.3   0.2   1:19.49 php-fpm8.3                                                                                                                                                            
 136989 root      20   0   49056  40360   7896 R  25.7   0.1   0:00.80 /usr/share/user                                                                                                                                                       
  48772 peerpoi+  20   0  796888  90348  29328 S  23.8   0.2   0:54.41 php-fpm8.4                                                                                                                                                            
 132828 alexand+  20   0  783596 189048 142612 S  23.8   0.4   0:21.39 php-fpm8.4                                                                                                                                                            
 136826 visitha+  20   0  700740  88112  50164 S  21.2   0.2   0:02.13 php-fpm8.4                                                                                                                                                            
 136898 thenetw+  20   0  676836  43196  28412 R  18.6   0.1   0:01.48 php-fpm8.4                                                                                                                                                            
 131956 antique+  20   0  874764 276208 138032 S  18.0   0.6   0:44.18 php-fpm8.4                                                                                                                                                            
  42104 singalo+  20   0  747616 100612  81932 R  17.7   0.2   3:40.37 php-fpm8.1                                                                                                                                                            
 136936 thenetw+  20   0  676836  43204  28408 R  17.7   0.1   0:00.99 php-fpm8.4                                                                                                                                                            
  69106 gybo      20   0  678592  92968  72828 R  17.4   0.2   4:09.79 php-fpm8.3                                                                                                                                                            
 135762 seofors+  20   0  695116 172084 139012 S  17.4   0.3   0:11.20 php-fpm8.4                                                                                                                                                            
 133147 visitha+  20   0  730216 118348  50676 S  16.4   0.2   0:28.86 php-fpm8.4                                                                                                                                                            
 136848 thenetw+  20   0  676680  43028  28412 R  15.8   0.1   0:01.75 php-fpm8.4                                                                                                                                                            
    490 root      19  -1  327444  57160  54872 R  15.4   0.1  23:19.73 systemd-journal                                                                                                                                                       
 135522 visitha+  20   0  800552 118608  54336 R  14.8   0.2   0:14.12 php-fpm8.4                                                                                                                                                            
   6108 lenovor+  20   0  763792 240036 208644 S  13.5   0.5   7:19.65 php-fpm8.3                                                                                                                                                            
  68997 gybo      20   0  753732  99944  78320 S  13.5   0.2   3:56.44 php-fpm8.3                                                                                                                                                            
 133018 seofors+  20   0  766984 172600 141348 R  12.9   0.3   0:29.18 php-fpm8.4                                                                                                                                                            
 136178 thenetw+  20   0  678728  45136  28408 S  12.9   0.1   0:05.98 php-fpm8.4                                                                                                                                                            
  95044 singalo+  20   0  746832 102680  84688 S   9.6   0.2   2:44.31 php-fpm8.1                                                                                                                                                            
   6404 root      20   0  123716 117408   7052 R   9.3   0.2   6:18.60 stats.pl                                                                                                                                                              
  60015 guide     20   0  674304 106492  88904 S   9.0   0.2   1:14.06 php-fpm8.3                                                                                                                                                            
  40336 lenovor+  20   0  763968 258652 225436 S   8.4   0.5   5:53.80 php-fpm8.3                                                                                                                                                            
  60172 guide     20   0  748852 112868  94540 S   8.4   0.2   1:14.99 php-fpm8.3                                                                                                                                                            
    756 syslog    20   0  222408   6856   4372 S   8.0   0.0  11:09.09 rsyslogd                                                                                                                                                              
 136637 thenetw+  20   0  678728  45112  28408 S   8.0   0.1   0:02.74 php-fpm8.4                                                                                                                                                            
 136568 thenetw+  20   0  678888  45264  28412 R   7.4   0.1   0:03.31 php-fpm8.4                                                                                                                                                            
 113578 opendkim  20   0  260476  11088   7172 S   6.8   0.0   0:51.75 opendkim                                                                                                                                                              
  32016 singalo+  20   0  746936 102380  84304 S   6.4   0.2   4:18.90 php-fpm8.1                                                                                                                                                            
 130193 www-data  20   0 3800968  40432   9140 S   6.4   0.1   0:19.78 apache2                                                                                                                                                               
 133477 seofors+  20   0  769580 174384 141472 S   6.1   0.4   0:24.08 php-fpm8.4                                                                                                                                                            
 130869 www-data  20   0 3801456  40880   8980 S   5.8   0.1   0:17.31 apache2                                                                                                                                                               
 135950 postfix   20   0   42164   8732   7776 R   4.5   0.0   0:00.70 cleanup                                                                                                                                                               
 120603 postfix   20   0   48024  12920  10728 S   3.9   0.0   0:02.87 smtp                                                                                                                                                                  
    885 root      20   0 3580632  77132  60572 S   3.5   0.2   5:10.08 fail2ban-server                                                                                                                                                       
 114930 postfix   20   0   48076  12964  10772 S   3.5   0.0   0:04.06 smtp                                                                                                                                                                  
 114927 postfix   20   0   48080  13076  10892 S   3.2   0.0   0:03.73 smtp 

Note: top shows %Cpu(s): 56.2, but Dashboard shows 88-94%

Can anyone suggest any possible issues/solutions?

TIA,

Paul

Edit/Addition - I have just noticed that no virtual memory is being used!!

Operating system Ubuntu Linux 22.04.5
Webmin version 2.660 Usermin version 2.560
Virtualmin version 8.1.0 Professional
Authentic theme version 26.60
Kernel and CPU Linux 5.15.0-190-generic on x86_64
Processor information Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 12 cores
Real memory 16.56 GiB used / 12.15 GiB cached / 47.04 GiB total
Virtual memory 0 bytes used / 3.99 GiB total
Local disk space 201.6 GiB used / 506.98 GiB free / 708.59 GiB total
Package updates All installed packages are up to date

You need to find out the full process name of those Webmin processes that are chewing a lot of CPU (type c in top to get a longer command field). /usr/share/webmin contains many, many, executable files, so without the full process name, there’s no way to know what’s actually chewing CPU. During backups, for example, Webmin load would be quite high.

I don’t know what systemd would be working so hard on. You’d probably need to do a pstree on that process to see what it’s spawning to know what it’s working on (systemd is the init system so everything is started by systemd). systemd is expected to be using a lot of CPU during boot, but pretty much at no other time, so it’s surprising to see it chewing more CPU than anything else.

Mariadb and php-fpm are expected to be working hard on a server with a website that has a lot of traffic. If you don’t have enough traffic to explain the load, it becomes a concern, and indicates poorly written applications or some kind of abuse.