| SYSTEM INFORMATION | |
|---|---|
| OS type and version | Debian Linux 12 |
| Webmin version | 2.670 |
| Usermin version | 2.570 |
| Virtualmin version | 8.2.0 GPL |
| Theme version | 26.70 |
| Nginx version | 1.22.1 |
| Package updates | All installed packages are up to date |
Hi all,
I have been troubleshooting a rather strange stability problem on one of my Virtualmin servers for quite some time now. At this point I am interested to hear whether anyone here has seen anything remotely similar, especially on older AMD Ryzen hardware, Debian 12, or in combination with Webmin/Virtualmin service management.
This has been a long investigation, so I will describe the full history from the beginning.
Original system
The server originally ran:
Debian 12
Virtualmin / Webmin
ASUS PRIME A320M-K
AMD Ryzen 3 PRO 2200G
32 GB DDR4-2400 (2 x 16 GB)
WD NVMe SSD
Additional SATA SSD storage
This machine had been running my hosting, mail, DNS and other services.
CPU upgrade and motherboard failure
Around the end of July I upgraded the CPU from the Ryzen 3 PRO 2200G to a Ryzen 7 1700X.
During the CPU/cooler work the plastic cooler retention mechanism also broke, which I temporarily worked around until the hardware could be sorted properly.
The important part is that the old ASUS A320 board then started behaving strangely. It began crashing before eventually failing completely.
I upgraded the ASUS BIOS from version 5409 to the latest available version. After the BIOS update it appeared to behave normally for roughly two days, but then the crashes returned and eventually the motherboard would no longer POST at all.
I tested the system with:
Ryzen 7 1700X
Ryzen 3 2200G
Different RAM modules
Two different PSUs
The board still would not POST, so I concluded that the ASUS motherboard itself had died.
I briefly obtained an ASRock A520M-HVS, but discovered that this board does not support either the Ryzen 7 1700X or Ryzen 3 2200G.
The final replacement became:
ASRock B450 Pro4 R2.0
AMD Ryzen 7 1700X
32 GB RAM
NVIDIA GeForce GT 710 for display
The important detail is that after replacing the motherboard, the server then ran for weeks without a single crash.
Problems appeared again later
The instability returned later.
One change that happened around the same period was that I fixed a LAN bottleneck. The server had effectively been operating around 100 Mbit/s before, while it is now connected at the full:
1000 Mb/s
Full Duplex
The onboard NIC is:
Realtek RTL8111/8168/8411
PCI ID 10ec:8168
Kernel driver: r8169
I do not claim that 1 Gbit Ethernet is the cause, but the timing is interesting because the system had been stable before that.
Nature of the crashes
These are not ordinary service crashes.
I have had several kinds of failures:
- Complete hard freezes
- Kernel panic
- RCU stalls
- Machine Check Exceptions
- A captured double fault / stack overflow
During the worst hard freezes:
SSH stops responding
Local keyboard stops responding
Num Lock does not respond
Magic SysRq does not respond
The display remains on the last screen or shows no useful new output
Only a physical reset works
So when this happens, the entire OS appears to be dead.
Machine Check Exceptions
Across different boots I have previously captured MCEs including:
CPU 0:
Machine Check: Bank 22
baa000000000080b
CPU 4:
Machine Check: Bank 0
baa0000000060185
CPU 13:
Machine Check: Bank 5
bea0000000000108
CPU 6:
Machine Check: Bank 5
bea0000000000108
CPU identification was:
PROCESSOR 2:800f11
These occurred on different logical CPUs and different machine-check banks.
RCU hard lock
I have also had a hard freeze with messages such as:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks
rcu_preempt kthread timer wakeup didn't happen
Possible timer handling issue on cpu=11
rcu_preempt kthread starved
That particular event originally made me suspicious of CPU 11 and networking IRQ processing.
NIC/IRQ investigation
At one point almost all Ethernet interrupt/softirq activity was ending up on CPU 11.
I therefore changed the NIC configuration so that the Realtek IRQ was assigned to CPU 2 and enabled RPS across multiple CPUs while avoiding the core containing CPUs 3/11.
The configuration worked correctly and I verified that the interrupts were actually being handled by CPU 2.
Unfortunately the server still crashed afterwards.
So the original CPU-11/IRQ-overload theory does not explain everything.
Temperatures and load
I installed a custom hardware logger that records, among other things:
CPU temperature
Available memory
Swap
System load
NIC IRQ counters
Recent kernel/hardware errors
Immediately before several crashes the machine was not under unusual stress.
For example, before one crash:
CPU Tdie around 40-46 C
20+ GB RAM available
0 swap usage
Load well below 1
No recent MCE logged
Therefore I do not see evidence of overheating, memory exhaustion, swapping or extreme server load causing these freezes.
BIOS testing
The replacement ASRock B450 Pro4 R2.0 originally ran BIOS P5.60.
I also tested disabling Global C-State Control.
That did not solve the problem.
On September 18 I upgraded the motherboard to:
ASRock BIOS P10.70
AGESA ComboAM4v2 1.2.0.12
The crashes still occurred afterwards, including two failures only a few hours apart.
So the newest BIOS did not solve it either.
Kdump
Because normal logs were not showing the cause, I configured kdump.
The crashkernel is currently:
crashkernel=512M
and I have also enabled:
kernel.panic_on_rcu_stall = 1
kernel.softlockup_panic = 1
kernel.hardlockup_panic = 1
Kdump successfully captured one particularly interesting crash.
Captured kernel panic
The dump was from Debian kernel:
6.1.0-53-amd64
Debian 6.1.187-1
The panic contained:
BUG: ENTRY_TRAMPOLINE stack guard page was hit
stack guard page
CPU: 4
PID: 38908
Comm: systemctl
RIP: error_entry+0xb5/0x130
recursive exc_page_fault frames
The crash analysis ultimately showed:
machine_kexec
__crash_kexec
crash_kexec
oops_end
handle_stack_overflow
exc_double_fault
asm_exc_double_fault
error_entry+181
So this was a double fault caused by an exception-stack overflow.
Why I am posting this on the Virtualmin forum
This part may be interesting to Virtualmin/Webmin users.
The process that happened to be executing when the kernel died was:
systemctl
PID 38908
Looking at the process context in the crash dump showed:
CWD: /usr/share/webmin/virtual-server
stdin: /dev/null
stdout: FIFO
stderr: /dev/null
fd 3: UNIX STREAM socket
So systemctl had been launched from a Virtualmin/Webmin context.
Further analysis of the exception frame showed that the process appeared to be in:
recvmsg()
on file descriptor 3, presumably communicating over a Unix socket with systemd/D-Bus.
To be clear: I am not saying Virtualmin caused the kernel crash.
A userspace systemctl command should obviously never be capable of causing a kernel double fault.
But because Virtualmin happened to be the process context when the kernel exception path exploded, I would be very interested if anyone has seen similar behaviour involving Virtualmin service operations or its monitoring scripts.
Safe RET / SRSO discovery
The Ryzen 7 1700X is a first-generation Zen CPU.
On kernel 6.1.187 the system was using:
Mitigation: safe RET
for Speculative Return Stack Overflow/SRSO mitigation.
When I analysed the exact machine code around the crash, the instruction at:
error_entry+0xb5
was a call to:
__handle_intr_saferet
In other words, the final fault happened precisely in the kernel’s Safe-RET exception/interrupt handling path.
Again, this does not necessarily mean __handle_intr_saferet itself caused the original corruption. It is possible that the exception stack was already in a bad state and this was simply where it finally overflowed.
Still, it was suspicious enough to test another SRSO mitigation.
I changed the kernel command line to:
spec_rstack_overflow=ibpb
Afterwards:
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow
reported:
Mitigation: IBPB
Interesting result from IBPB
Before this change, some crashes occurred after only approximately:
2 hours 13 minutes
3 hours 43 minutes
With IBPB enabled, the server then remained stable for almost six days.
That looked extremely promising.
Unfortunately on September 25 it hard-froze again.
This time:
No SSH
No keyboard response
No SysRq
No new panic visible
No new vmcore
So although changing Safe RET to IBPB appears to have made a significant difference, it did not completely eliminate the underlying instability.
My current interpretation is therefore that the captured Safe-RET crash was real and relevant, but may not be the only failure mechanism.
Separate kdump/networking problem
After the hard reset on September 25, networking did not automatically come back.
I found this in the boot log:
kdump-tools.service: Found ordering cycle on networking.service/start
kdump-tools.service:
Job networking.service/start deleted to break ordering cycle
This was unrelated to the original hard freeze, but it meant the server booted without its static IPv4 configuration.
The normal configuration is:
auto lo enp9s0
iface lo inet loopback
iface enp9s0 inet static
address 192.168.1.28
netmask 255.255.255.0
network 192.168.1.0
gateway 192.168.1.1
I found that Debian’s kdump-tools.service contained:
After=local-fs.target
After=networking.service
Before=basic.target
which created the ordering cycle on this system.
I removed the After=networking.service dependency from the local systemd unit.
Networking now starts correctly and kdump still reports:
current state : ready to kdump
So this was a separate problem introduced during the crash-debugging setup, not the original instability.
Driver and firmware checks
I also checked whether I simply had missing chipset or hardware drivers.
Everything important appears to have a driver:
NVMe -> nvme
AMD/ASMedia SATA -> ahci
USB -> xhci_hcd
Realtek NIC -> r8169
NVIDIA GT710 -> nvidia
AMD microcode is installed and currently loaded:
amd64-microcode 3.20250311.1~deb12u1
microcode patch level 0x8001139
I do not see missing firmware errors for the important hardware.
Now testing Debian kernel 6.12
Instead of going backwards to an older 6.1 kernel, I have now moved forward to Debian’s newer 6.12 kernel.
The server is currently running:
Linux somehostname
6.12.107+deb12-amd64
Debian 6.12.107-1~deb12u1
The kernel command line still contains:
spec_rstack_overflow=ibpb
crashkernel=512M
and SRSO currently reports:
Mitigation: IBPB
Current networking:
enp9s0
192.168.1.28/24
1000 Mb/s
Full Duplex
Link detected: yes
networking.service: active
The NVIDIA Tesla 470 driver successfully built through DKMS for 6.12:
nvidia-tesla-470/470.256.02, 6.12.107+deb12-amd64: installed
and nvidia-smi works normally.
Kdump also survived the kernel upgrade:
current state : ready to kdump
kexec_crash_loaded = 1
The only NVIDIA messages currently classified as errors are from nvidia-persistenced:
NUMA: Failed ioctl call to get device NUMA Info: Invalid argument
device 0000:0a:00.0 - NUMA: Failed to get device NUMA info
device 0000:0a:00.0 - failed to online memory
The actual GPU driver is nevertheless loaded and nvidia-smi works.
Current hardware/software summary
Server:
ASRock B450 Pro4 R2.0
AMD Ryzen 7 1700X
32 GB RAM
NVIDIA GeForce GT 710
Realtek RTL8111/8168/8411 onboard Ethernet
OS:
Debian 12
Current kernel:
6.12.107+deb12-amd64
Debian 6.12.107-1~deb12u1
Hosting stack:
Virtualmin / Webmin
Nginx
Postfix
Dovecot
BIND
etc.
Network:
1 Gbit/s Full Duplex
r8169
CPU microcode:
0x8001139
SRSO mitigation:
IBPB
Kdump:
enabled and loaded
512 MB crashkernel
What I am trying to determine
At this stage I see several possibilities:
- An underlying Ryzen 1700X / platform instability which occasionally manifests as MCEs or total lockups.
- A kernel regression or interaction affecting first-generation Zen.
- More than one issue, with Safe RET responsible for the captured double-fault panic but something else responsible for the complete hard freezes.
- A PCIe/DMA/interrupt-related issue that became easier to trigger once the NIC began running continuously at 1 Gbit/s.
- Some interaction I have not considered involving the Realtek r8169 driver or another PCIe device.
- Much less likely, something Virtualmin/Webmin happens to trigger through service management, although I would expect a kernel bug to be required for userspace activity to produce this sort of failure.
I am currently leaving the hardware configuration unchanged and testing kernel 6.12 to see whether the behaviour changes.
I would especially be interested to hear from anyone who has experience with:
- Virtualmin/Webmin on first-generation Ryzen
- hard freezes on Ryzen 1000 / Summit Ridge under Debian
- MCE Bank 5
bea0000000000108 ENTRY_TRAMPOLINE stack guard page was hiterror_entry/__handle_intr_saferet- Safe RET versus IBPB on Zen 1
- r8169 under sustained Gigabit traffic
- hard freezes where even the NMI watchdog and kdump cannot execute
- Virtualmin launching
systemctlfrom/usr/share/webmin/virtual-server - similar behaviour after recent Debian 6.1 kernel updates
At this point I am mainly looking for other people’s experiences or anything obvious I may have overlooked.
Thanks in advance for any ideas.