Last update/reboot made my VPS inaccessible

Virtualmin
1

I’ve my VPSs hosted by Contabo. And I do most of the coming Webmin updates on time. The last update required a reboot. After I clicked to be rebooted, the VPS never came alive again. Not even accessible by SSH.

This is my second VPS having this problem. I reported to Contabo support but since yesterday I’m still waiting for their reply.

Anyone could explain what happened and how to solve it, please. I’m scared to reboot my other VPSs.

Update: I just got a reply from Contabo support “Your /boot file system is likely full and the system was not able to generate initramfs for newer kernels. There are currently three kernels installed on your system, of which the newer two (94 and 92) failed to boot. We have booted the kernel ending in 91.
Please make some space on your /boot and regenerate the initramfs for the newest kernel.”
But I checked my boot folder: /boot 82% Free (751.85 MiB) 921.99 MiB Total. I guess there is a problem with the update process?

SYSTEM INFORMATION
OS type and version Ubuntu 18.04 (and the 1st VPS has Ubuntu 20.04)

Webmin/Virtualmin updates log:

Reading package lists…
Building dependency tree…
Reading state information…
Suggested packages:
default-dbus-session-bus | dbus-session-bus glibc-doc systemd-container
The following packages will be upgraded:
curl dbus libc-bin libc-dev-bin libc6 libc6-dev libcurl3-gnutls libcurl4
libdbus-1-3 libnss-systemd libpam-systemd libsystemd0 libudev1
linux-libc-dev locales systemd systemd-sysv systemd-timesyncd udev
19 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.7 MB of archives.
After this operation, 26.6 kB of additional disk space will be used.
Get:1 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libc6-dev amd64 2.31-0ubuntu9.9 [2519 kB]
Get:2 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libc-dev-bin amd64 2.31-0ubuntu9.9 [71.8 kB]
Get:3 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-110.124 [1115 kB]
Get:4 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libc6 amd64 2.31-0ubuntu9.9 [2722 kB]
Get:5 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 locales all 2.31-0ubuntu9.9 [3869 kB]
Get:6 Index of /ubuntu focal-security/main amd64 curl amd64 7.68.0-1ubuntu2.11 [162 kB]
Get:7 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libc-bin amd64 2.31-0ubuntu9.9 [633 kB]
Get:8 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 udev amd64 245.4-4ubuntu3.17 [1364 kB]
Get:9 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libudev1 amd64 245.4-4ubuntu3.17 [76.5 kB]
Get:10 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 systemd-sysv amd64 245.4-4ubuntu3.17 [10.3 kB]
Get:11 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 systemd-timesyncd amd64 245.4-4ubuntu3.17 [28.1 kB]
Get:12 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libnss-systemd amd64 245.4-4ubuntu3.17 [96.0 kB]
Get:13 Index of /ubuntu focal-security/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.11 [235 kB]
Get:14 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 dbus amd64 1.12.16-2ubuntu2.2 [151 kB]
Get:15 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libdbus-1-3 amd64 1.12.16-2ubuntu2.2 [179 kB]
Get:16 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libpam-systemd amd64 245.4-4ubuntu3.17 [186 kB]
Get:17 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 systemd amd64 245.4-4ubuntu3.17 [3809 kB]
Get:18 Index of /ubuntu focal-security/main amd64 libcurl3-gnutls amd64 7.68.0-1ubuntu2.11 [233 kB]
Get:19 http://asi-fs-n.contabo.net/ubuntu focal-updates/main amd64 libsystemd0 amd64 245.4-4ubuntu3.17 [269 kB]
[master a2e61d4] saving uncommitted changes in /etc prior to apt run
13 files changed, 268 insertions(+), 7 deletions(-)
create mode 100644 letsencrypt/archive/web.mysite.cc/cert12.pem
create mode 100644 letsencrypt/archive/web.mysite.cc/chain12.pem
create mode 100644 letsencrypt/archive/web.mysite.cc/fullchain12.pem
create mode 100644 letsencrypt/archive/web.mysite.cc/privkey12.pem
create mode 100644 letsencrypt/csr/0918_csr-certbot.pem
create mode 100644 letsencrypt/keys/0918_key-certbot.pem
Preconfiguring packages …
Fetched 17.7 MB in 1s (33.1 MB/s)
(Reading database …
(Reading database … 5%
(Reading database … 10%
(Reading database … 15%
(Reading database … 20%
(Reading database … 25%
(Reading database … 30%
(Reading database … 35%
(Reading database … 40%
(Reading database … 45%
(Reading database … 50%
(Reading database … 55%
(Reading database … 60%
(Reading database … 65%
(Reading database … 70%
(Reading database … 75%
(Reading database … 80%
(Reading database … 85%
(Reading database … 90%
(Reading database … 95%
(Reading database … 100%
(Reading database … 196438 files and directories currently installed.)
Preparing to unpack …/libc6-dev_2.31-0ubuntu9.9_amd64.deb …
Unpacking libc6-dev:amd64 (2.31-0ubuntu9.9) over (2.31-0ubuntu9.7) …
Preparing to unpack …/libc-dev-bin_2.31-0ubuntu9.9_amd64.deb …
Unpacking libc-dev-bin (2.31-0ubuntu9.9) over (2.31-0ubuntu9.7) …
Preparing to unpack …/linux-libc-dev_5.4.0-110.124_amd64.deb …
Unpacking linux-libc-dev:amd64 (5.4.0-110.124) over (5.4.0-109.123) …
Preparing to unpack …/libc6_2.31-0ubuntu9.9_amd64.deb …
Unpacking libc6:amd64 (2.31-0ubuntu9.9) over (2.31-0ubuntu9.7) …
Setting up libc6:amd64 (2.31-0ubuntu9.9) …
(Reading database …
(Reading database … 5%
(Reading database … 10%
(Reading database … 15%
(Reading database … 20%
(Reading database … 25%
(Reading database … 30%
(Reading database … 35%
(Reading database … 40%
(Reading database … 45%
(Reading database … 50%
(Reading database … 55%
(Reading database … 60%
(Reading database … 65%
(Reading database … 70%
(Reading database … 75%
(Reading database … 80%
(Reading database … 85%
(Reading database … 90%
(Reading database … 95%
(Reading database … 100%
(Reading database … 196438 files and directories currently installed.)
Preparing to unpack …/locales_2.31-0ubuntu9.9_all.deb …
Unpacking locales (2.31-0ubuntu9.9) over (2.31-0ubuntu9.7) …
Preparing to unpack …/libc-bin_2.31-0ubuntu9.9_amd64.deb …
Unpacking libc-bin (2.31-0ubuntu9.9) over (2.31-0ubuntu9.7) …
Setting up libc-bin (2.31-0ubuntu9.9) …
(Reading database …
(Reading database … 5%
(Reading database … 10%
(Reading database … 15%
(Reading database … 20%
(Reading database … 25%
(Reading database … 30%
(Reading database … 35%
(Reading database … 40%
(Reading database … 45%
(Reading database … 50%
(Reading database … 55%
(Reading database … 60%
(Reading database … 65%
(Reading database … 70%
(Reading database … 75%
(Reading database … 80%
(Reading database … 85%
(Reading database … 90%
(Reading database … 95%
(Reading database … 100%
(Reading database … 196438 files and directories currently installed.)
Preparing to unpack …/udev_245.4-4ubuntu3.17_amd64.deb …
Unpacking udev (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/libudev1_245.4-4ubuntu3.17_amd64.deb …
Unpacking libudev1:amd64 (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Setting up libudev1:amd64 (245.4-4ubuntu3.17) …
(Reading database …
(Reading database … 5%
(Reading database … 10%
(Reading database … 15%
(Reading database … 20%
(Reading database … 25%
(Reading database … 30%
(Reading database … 35%
(Reading database … 40%
(Reading database … 45%
(Reading database … 50%
(Reading database … 55%
(Reading database … 60%
(Reading database … 65%
(Reading database … 70%
(Reading database … 75%
(Reading database … 80%
(Reading database … 85%
(Reading database … 90%
(Reading database … 95%
(Reading database … 100%
(Reading database … 196438 files and directories currently installed.)
Preparing to unpack …/0-systemd-sysv_245.4-4ubuntu3.17_amd64.deb …
Unpacking systemd-sysv (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/1-systemd-timesyncd_245.4-4ubuntu3.17_amd64.deb …
Unpacking systemd-timesyncd (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/2-libnss-systemd_245.4-4ubuntu3.17_amd64.deb …
Unpacking libnss-systemd:amd64 (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/3-dbus_1.12.16-2ubuntu2.2_amd64.deb …
Unpacking dbus (1.12.16-2ubuntu2.2) over (1.12.16-2ubuntu2.1) …
Preparing to unpack …/4-libdbus-1-3_1.12.16-2ubuntu2.2_amd64.deb …
Unpacking libdbus-1-3:amd64 (1.12.16-2ubuntu2.2) over (1.12.16-2ubuntu2.1) …
Preparing to unpack …/5-libpam-systemd_245.4-4ubuntu3.17_amd64.deb …
Unpacking libpam-systemd:amd64 (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/6-systemd_245.4-4ubuntu3.17_amd64.deb …
Unpacking systemd (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Preparing to unpack …/7-libsystemd0_245.4-4ubuntu3.17_amd64.deb …
Unpacking libsystemd0:amd64 (245.4-4ubuntu3.17) over (245.4-4ubuntu3.16) …
Setting up libsystemd0:amd64 (245.4-4ubuntu3.17) …
(Reading database …
(Reading database … 5%
(Reading database … 10%
(Reading database … 15%
(Reading database … 20%
(Reading database … 25%
(Reading database … 30%
(Reading database … 35%
(Reading database … 40%
(Reading database … 45%
(Reading database … 50%
(Reading database … 55%
(Reading database … 60%
(Reading database … 65%
(Reading database … 70%
(Reading database … 75%
(Reading database … 80%
(Reading database … 85%
(Reading database … 90%
(Reading database … 95%
(Reading database … 100%
(Reading database … 196438 files and directories currently installed.)
Preparing to unpack …/curl_7.68.0-1ubuntu2.11_amd64.deb …
Unpacking curl (7.68.0-1ubuntu2.11) over (7.68.0-1ubuntu2.10) …
Preparing to unpack …/libcurl4_7.68.0-1ubuntu2.11_amd64.deb …
Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.11) over (7.68.0-1ubuntu2.10) …
Preparing to unpack …/libcurl3-gnutls_7.68.0-1ubuntu2.11_amd64.deb …
Unpacking libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.11) over (7.68.0-1ubuntu2.10) …
Setting up linux-libc-dev:amd64 (5.4.0-110.124) …
Setting up libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.11) …
Setting up locales (2.31-0ubuntu9.9) …
Generating locales (this might take a while)…
en_AG.UTF-8… done
en_AU.UTF-8… done
en_BW.UTF-8… done
en_CA.UTF-8… done
en_DK.UTF-8… done
en_GB.UTF-8… done
en_HK.UTF-8… done
en_IE.UTF-8… done
en_IL.UTF-8… done
en_IN.UTF-8… done
en_NG.UTF-8… done
en_NZ.UTF-8… done
en_PH.UTF-8… done
en_SG.UTF-8… done
en_US.UTF-8… done
en_ZA.UTF-8… done
en_ZM.UTF-8… done
en_ZW.UTF-8… done
Generation complete.
Setting up libdbus-1-3:amd64 (1.12.16-2ubuntu2.2) …
Setting up dbus (1.12.16-2ubuntu2.2) …
A reboot is required to replace the running dbus-daemon.
Please reboot the system when convenient.
Setting up udev (245.4-4ubuntu3.17) …
update-initramfs: deferring update (trigger activated)
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.11) …
Setting up curl (7.68.0-1ubuntu2.11) …
Setting up libc-dev-bin (2.31-0ubuntu9.9) …
Setting up libc6-dev:amd64 (2.31-0ubuntu9.9) …
Setting up systemd (245.4-4ubuntu3.17) …
Setting up systemd-timesyncd (245.4-4ubuntu3.17) …
Setting up systemd-sysv (245.4-4ubuntu3.17) …
Setting up libnss-systemd:amd64 (245.4-4ubuntu3.17) …
Setting up libpam-systemd:amd64 (245.4-4ubuntu3.17) …
Processing triggers for libc-bin (2.31-0ubuntu9.9) …
Processing triggers for man-db (2.9.1-1) …
Processing triggers for initramfs-tools (0.136ubuntu6.7) …
update-initramfs: Generating /boot/initrd.img-5.4.0-91-generic

2

Please read search first the forum guidelines! :wink:
Forum Guidelines: Please read before posting! - Help! (Home for newbies) - Virtualmin Community

And check ask contabo for those and the kernels on it if they can see know they are booting from

Also ofcourse you need to know after which updates this is, and did you do those at the VPS’s

You have snapshots at Contabo? Yes but only “short” live before delettion from those, check how much snapshots and snapshot times.

Also before rebooting make good ofline other location working backups.

3

Thank you for your reply. Sure I tried searching before I posted but there were no results.

Yes, I do snapshots frequently. This happened with the last Virtualmin update. Previous updates never asked for a reboot nor did I have a problem with rebooting before.

I still believe there is a problem in this update as I faced the same results on 3 VPSs so far and I still don’t know what to do with the rest of my VPSs, I’m worried if I reboot, they will be no boot.

4

Those aren’t Virtualmin updates, you’re showing system updates above…nothing we ship would ever touch initramfs. Virtualmin shows you available system updates, but they are not “Virtualmin updates”. It’s the same as if you ran apt-get to install the listed packages.

5

In my opinion that’s a very wrong approach on your vendor’s side on isntalling OS’s /boot in seperate partition causing /boot to be filled in case you’ll not remove old kernels.

I’ve faced this issue many times and fixed the same for multiple clients as well.

All you’ll need to do before reboot is check if there are any kernels autoremovable with following command

sudo apt autoremove

Press y to confirm if you’re satisfied with autoremoval packages if not simply press N and install the removal packages maunally to mark them as not auto removable.

I’ve tried to switch all my clients to AWS as they follow simple installation with all in one partitions

i.e. / which will avoid all these issues by default.

6

Thank you @Joe for the clarification.

For the rest of my VPSs, I did all the updates but I didn’t reboot them. Then I contacted Contabo to do the rebooting and fix if any problems would arise.

Their reply was: “initramfs was absent in new ones. I had to create them manually with commands, I´ve restarted them to the new kernel.”

I’m sharing this hoping to find the reason for what I faced in all of my 7 VPSs and how to prevent it from happening in the future.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.