Installing Virtualmin on Debian 12 crashes with Out of Memory error

panos72 · December 12, 2024, 12:59pm

SYSTEM INFORMATION
OS type and version	Linux Debian 12
Virtualmin version	7.5.2 GPL

I have a fresh install of latest Debian 12 on a VM with 2 CPU cores and 16GB RAM. After the installation I did the following changes:

Set static IP
Disable ipv6
Allow remote root ssh
Installed iptables and set up basic rules
Removed Apache2 (was included with initial OS installation).

Then I run through SSH

sudo sh -c "$(curl -fsSL https://software.virtualmin.com/gpl/scripts/virtualmin-install.sh)" -- --bundle LAMP

as described on Virtualmin’s site.

When it reaches phase 3, “Installing Virtualmin 7 and all related packages” it takes some time working and then it crashes. Console gives the message

Out of memory: Killed process 113167 (/usr/share/webm) total-vm:32810828kB, anon-rss:32018772kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:64248kB oom_score_adj:0

I restored a snapshot of the server (just before executing the Virtualmin script), increased VMs RAM to 32GB and run the script again, with the same result.

dmesg lines for oom killer:

[ 3053.465963] systemd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[ 3053.465972] CPU: 1 PID: 1 Comm: systemd Not tainted 6.1.0-28-amd64 #1  Debian 6.1.119-1
[ 3053.465975] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
[ 3053.465976] Call Trace:
[ 3053.466008]  <TASK>
[ 3053.466011]  dump_stack_lvl+0x44/0x5c
[ 3053.466017]  dump_header+0x4a/0x211
[ 3053.466021]  oom_kill_process.cold+0xb/0x10
[ 3053.466024]  out_of_memory+0x1fd/0x4c0
[ 3053.466029]  __alloc_pages_slowpath.constprop.0+0xc83/0xe40
[ 3053.466034]  __alloc_pages+0x305/0x330
[ 3053.466036]  folio_alloc+0x17/0x50
[ 3053.466039]  __filemap_get_folio+0x155/0x340
[ 3053.466044]  filemap_fault+0x139/0x910
[ 3053.466047]  ? filemap_map_pages+0x153/0x700
[ 3053.466050]  __do_fault+0x33/0x110
[ 3053.466053]  do_fault+0x1b9/0x410
[ 3053.466057]  __handle_mm_fault+0x660/0xfa0
[ 3053.466060]  ? __blk_mq_free_request+0x71/0xe0
[ 3053.466064]  handle_mm_fault+0xdb/0x2d0
[ 3053.466067]  do_user_addr_fault+0x191/0x550
[ 3053.466070]  exc_page_fault+0x70/0x170
[ 3053.466074]  asm_exc_page_fault+0x22/0x30
[ 3053.466077] RIP: 0033:0x7fca3c1ccee7
[ 3053.466082] Code: Unable to access opcode bytes at 0x7fca3c1ccebd.
[ 3053.466083] RSP: 002b:00007ffc159024e0 EFLAGS: 00010202
[ 3053.466085] RAX: 0000000000000000 RBX: 00005602bbc6a130 RCX: 00000000000001a1
[ 3053.466086] RDX: 00005602bbbbaca0 RSI: 0000000000000000 RDI: 0000000000000007
[ 3053.466087] RBP: 00005602bbc6a2d0 R08: 0000000000000007 R09: 00005602bbbbab10
[ 3053.466088] R10: 60c14890e4436c3e R11: 00007ffc15902580 R12: 0000000000000000
[ 3053.466089] R13: 00005602bbc6a2d0 R14: 0000000000000000 R15: 0000000000000001
[ 3053.466092]  </TASK>
[ 3053.466097] Mem-Info:
[ 3053.466099] active_anon:9388 inactive_anon:8085129 isolated_anon:0
                active_file:49 inactive_file:95 isolated_file:0
                unevictable:0 dirty:0 writeback:0
                slab_reclaimable:26993 slab_unreclaimable:11090
                mapped:705 shmem:729 pagetables:17254
                sec_pagetables:0 bounce:0
                kernel_misc_reclaimable:0
                free:48773 free_pcp:15 free_cma:0
[ 3053.466103] Node 0 active_anon:37552kB inactive_anon:32340516kB active_file:196kB inactive_file:380kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2820kB dirty:0kB writeback:0kB shmem:2916kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1026048kB writeback_tmp:0kB kernel_stack:4432kB pagetables:69016kB sec_pagetables:0kB all_unreclaimable? no
[ 3053.466107] Node 0 DMA free:11264kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 3053.466112] lowmem_reserve[]: 0 2959 32042 32042 32042
[ 3053.466114] Node 0 DMA32 free:122480kB boost:0kB min:6236kB low:9264kB high:12292kB reserved_highatomic:0KB active_anon:0kB inactive_anon:2931360kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:3063680kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 3053.466118] lowmem_reserve[]: 0 0 29082 29082 29082
[ 3053.466121] Node 0 Normal free:61348kB boost:2048kB min:63360kB low:93140kB high:122920kB reserved_highatomic:0KB active_anon:37552kB inactive_anon:29409156kB active_file:152kB inactive_file:1084kB unevictable:0kB writepending:0kB present:30408704kB managed:29788496kB mlocked:0kB bounce:0kB free_pcp:108kB local_pcp:0kB free_cma:0kB
[ 3053.466125] lowmem_reserve[]: 0 0 0 0 0
[ 3053.466127] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
[ 3053.466140] Node 0 DMA32: 0*4kB 2*8kB (U) 2*16kB (U) 4*32kB (UM) 3*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 1*1024kB (M) 1*2048kB (U) 29*4096kB (M) = 122480kB
[ 3053.466152] Node 0 Normal: 1628*4kB (UME) 1279*8kB (UME) 759*16kB (UME) 400*32kB (UME) 174*64kB (UME) 70*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61784kB
[ 3053.466162] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 3053.466163] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 3053.466164] 6374 total pagecache pages
[ 3053.466165] 5477 pages in swap cache
[ 3053.466166] Free swap  = 0kB
[ 3053.466166] Total swap = 999420kB
[ 3053.466167] 8388478 pages RAM
[ 3053.466168] 0 pages HighMem/MovableOnly
[ 3053.466168] 171594 pages reserved
[ 3053.466169] 0 pages hwpoisoned
[ 3053.466169] Tasks state (memory values in pages):
[ 3053.466170] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 3053.466173] [    261]     0   261     8301      604    94208      240          -250 systemd-journal
[ 3053.466176] [    290]     0   290     6713      499    73728        0         -1000 systemd-udevd
[ 3053.466178] [    523]     0   523    13090      389    94208       11             0 VGAuthService
[ 3053.466181] [    526]     0   526    41919      644    90112      242             0 vmtoolsd
[ 3053.466182] [    527]     0   527     1652       60    53248        3             0 cron
[ 3053.466184] [    529]   100   529     2341      195    57344        0          -900 dbus-daemon
[ 3053.466186] [    531]     0   531     4304      256    77824        0             0 systemd-logind
[ 3053.466188] [    648]     0   648     1530      130    53248        0             0 login
[ 3053.466190] [    661]     0   661     3858      289    65536       22         -1000 sshd
[ 3053.466192] [    801]     0   801     4717      419    77824        0           100 systemd
[ 3053.466193] [    802]     0   802    42194      357    86016      412           100 (sd-pam)
[ 3053.466195] [    808]     0   808     1999      341    57344        0             0 bash
[ 3053.466197] [    822]     0   822     4497      406    73728       62             0 sshd
[ 3053.466198] [    828]     0   828     2003       11    57344      350             0 bash
[ 3053.466200] [    889]     0   889      723       22    40960       99             0 sh
[ 3053.466202] [    985]   997   985    22521      215    73728        4             0 systemd-timesyn
[ 3053.466204] [   2262]     0  2262    20314        6    61440      583             0 gpg-agent
[ 3053.466206] [   5109]     0  5109    26144    16526   249856     5521             0 perl
[ 3053.466207] [   6334]     0  6334     9441     1157   102400     4228             0 miniserv.pl
[ 3053.466210] [  91707]     0 91707    10665       75    73728       85             0 master
[ 3053.466212] [  91708]   103 91708    10763      167    77824        0             0 pickup
[ 3053.466213] [  91709]   103 91709    10774      168    73728        0             0 qmgr
[ 3053.466215] [  91919]   104 91919    36987      444   188416     7697             0 named
[ 3053.466217] [  99560]   105 99560    41008      812   151552      164             0 freshclam
[ 3053.466218] [ 107069]     0 107069     4952      267    61440      165             0 proftpd
[ 3053.466222] [ 108100]   102 108100   318818    18645   622592    26638             0 mariadbd
[ 3053.466224] [ 108267]   111 108267    32425      255   102400      126             0 opendkim
[ 3053.466225] [ 110728]   112 110728    20314        7    65536       76             0 gpg-agent
[ 3053.466227] [ 111115]     0 111115    47801    29645   368640     3253             0 perl
[ 3053.466229] [ 112638]     0 112638    47801    29374   352256     3525             0 spamd child
[ 3053.466230] [ 112639]     0 112639    47801    29488   352256     3411             0 spamd child
[ 3053.466232] [ 112841]     0 112841    32602     5504   155648      535             0 firewalld
[ 3053.466234] [ 112910]   996 112910    58687      227    86016        0             0 polkitd
[ 3053.466238] [ 112984]     0 112984     2056      104    57344       15             0 dovecot
[ 3053.466239] [ 112991]   107 112991     1186       41    49152        0             0 anvil
[ 3053.466241] [ 112992]     0 112992     1220       79    49152        0             0 log
[ 3053.466243] [ 112993]     0 112993     1844      319    57344        0             0 config
[ 3053.466244] [ 113005]     0 113005    66704     2579   233472      176             0 php-fpm8.2
[ 3053.466246] [ 113006]    33 113006    66829     1967   204800      173             0 php-fpm8.2
[ 3053.466248] [ 113007]    33 113007    66829     1964   204800      176             0 php-fpm8.2
[ 3053.466249] [ 113167]     0 113167  8202707  8004693 65789952   195378             0 /usr/share/webm
[ 3053.466251] [ 113170]     0 113170    10364      214   110592     7526             0 /usr/share/webm
[ 3053.466253] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=/usr/share/webm,pid=113167,uid=0
[ 3053.466268] Out of memory: Killed process 113167 (/usr/share/webm) total-vm:32810828kB, anon-rss:32018772kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:64248kB oom_score_adj:0

Screenshots:

The ssh session
When I saw the script crashing again, I gave many times the “free” command. You can see the results in the second screenshot.

Any help would be really appreciated.
Thank you!

Ilia · December 12, 2024, 6:16pm

Can you reproduce the same issue on the clean Debian 12 Server instance?

To try to work around this issue:

Download virtualmin-install.sh script and run it with --setup parameter—this will setup Virtualmin repos for your system
Install Webmin manually by running apt-get install webmin
Afterwards, stop Webmin service using systemctl stop webmin
Perform Virtualmin installation using virtualmin-install.sh script
Start Webmin afterwards using systemctl start webmin

Webmin logs can be found in /var/webmin directory.

Joe · December 12, 2024, 11:00pm

This looks like something pathological is happening. Some postinstall thing is going crazy.

Ideally, we could see an strace of that process before it runs away too much and maybe lsof -p <pid> of the big/growing process. How long does it take for things to get out of hand and the OOM killer to kick in?

Ilia · December 12, 2024, 11:35pm

@panos72 What is the output of:

cat /etc/os-release

panos72 · December 13, 2024, 6:10am

Thank you for your answers!

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Now I’ll try starting from scratch without Apache and see what happens

panos72 · December 13, 2024, 6:23am

The “Installing Virtualmin 7…” process takes several minutes. But “playing” with the “free” command (see screenshot 2) it seems that when the problem appears, it escalates pretty fast and it takes less than 2 minutes to return the OOM message. The results seen on the screenshot are within less than a minute.

tpnsolutions · December 13, 2024, 7:46am

@panos72,

Who is the hosting provider, and why “type” of VPS is this being installed on?

I run Debian 12 based Virtualmin installs at Digital Ocean with far less resources then you noted, and have run into NO issues.

I wonder if this is one of those “over sold” providers whereby they advertise higher resource availability, but in reality are throttling those resources whereby a busy server could cause all VPS’ to run out of resources during peak hours. *** Not all VPS providers are equally setup ***

Joe · December 13, 2024, 8:14am

No, I think we’re seeing some Webmin process grow extravagantly, I don’t think it looks like an oversold situation, at least none I’ve seen. It’s unusual, but I don’t think we can blame the host.

panos72 · December 13, 2024, 10:47am

It worked!
I installed Debian from scratch without Apache. Also resized the swap partition to 6GB (from Debian’s 1GB default).

The first attempt failed but not with OOM error. It seems that virtualmin resets iptables rules but not policies. So with INPUT policy “DROP” but the “accept ssh” rule deleted, my ssh session died.

Finally I managed to install Virtualmin from console. Later I realized that it would be easier to leave INPUT policy to “accept”.

Anyway, job is done. Thank you all for your answers!

tpnsolutions · December 13, 2024, 3:56pm

@panos72,

Extremely odd, I install Virtualmin without issues on a system with 2GB of RAM available, no ommissions of software and never run into an issue where it won’t complete the install.

Further, FirewallD is typically installed by default on my Debian 12 VPS’ with appropriate firewall rules being created for all web ports required.

*** FirewallD is an alternative interface to the older “iptables”, check for FirewallD under Networking as it may have been configured in place of “iptables”. ***

system · December 21, 2024, 3:56pm

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.