I am having trouble updating a server from Debian 10 to Debian 12. I also tried updating from Debian 10 to 11 and both fail.
Via SSH I apt update - apt upgrade and all is current.
If I do apt full-upgrade the disk gets full and errors out saying there is no space. Before I start the update there is 30+GB available from a 55GB disk.
Above is a screenshot of the Virtualmin dashboard.
I have attempted to upgrade this server 4 times and have never been able to complete the upgrade.
I don’t know if this is important, but last night I ran System Settings | Re-Check Configuration and the first time I got impatient and force reloaded the page after 10 minutes. Later in the evening I tried running it again and after 45 minutes again force reloaded the page. Re-Check Configuration never completes.
Seems like you need to free up some disk. Either deleting stuff or temporarily moving it off. How much free space do you have before starting the upgrade?
The original post mentions 30gb free space which i thought should have been enough unless the boot partition is full and can not install the new kernel perhaps
Please understand this is the BEFORE update. Each of the 4 times I attempted updating this server the drive got full and the system crashed and required restoring from a backup. This df -h is therefore the results of the drive before any update.
As you can see there is 35GB available. After running apt update, apt upgrade, changing the sources.list to the bookworm repositories and again running apt update then apt upgrade all 35GB is consumed, the upgrade fails with zero space available.
@tpnsolutions - I used your link to run another attempt to upgrade. Here are the steps I did and some of the errors I encountered. As I write this the server is borked and none of the websites work.
Update Procedure Notes Debian 10-11:
apt update
apt upgrade
apt autoremove
apt autoclean
apt purge
nano /etc/apt/sources.list /etc/apt/sources.list.d/*
- Rename repositories from buster to bullseye
apt update
Err:4 https://packages.sury.org/php bullseye InRelease
The following signatures were invalid: EXPKEYSIG B188E2B695BD4743 DEB.SURY.ORG Automatic Signing Key <deb@sury.org>
apt-key adv --fetch-keys https://packages.sury.org/php/apt.gpg
apt update
Err:6 http://software.virtualmin.com/vm/6/gpl/apt virtualmin-bullseye InRelease
Cannot initiate the connection to software.virtualmin.com:80 (2001:bc8:608:b01::1). - connect (101: Network is unreachable) Could not connect to software.virtualmin.com:80 (51.158.66.131), connection timed out
Err:7 http://software.virtualmin.com/vm/6/gpl/apt virtualmin-universal InRelease
Cannot initiate the connection to software.virtualmin.com:80 (2001:bc8:608:b01::1). - connect (101: Network is unreachable)
Fetched 263 kB in 30s (8,682 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
729 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://software.virtualmin.com/vm/6/gpl/apt/dists/virtualmin-bullseye/InRelease Cannot initiate the connection to software.virtualmin.com:80 (2001:bc8:608:b01::1). - connect (101: Network is unreachable) Could not connect to software.virtualmin.com:80 (51.158.66.131), connection timed out
W: Failed to fetch http://software.virtualmin.com/vm/6/gpl/apt/dists/virtualmin-universal/InRelease Cannot initiate the connection to software.virtualmin.com:80 (2001:bc8:608:b01::1). - connect (101: Network is unreachable)
W: Some index files failed to download. They have been ignored, or old ones used instead.
virtualmin setup-repos
apt update
apt upgrade --without-new-pkgs
df -h
root@universe:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 983M 0 983M 0% /dev
tmpfs 200M 21M 179M 11% /run
/dev/vda1 52G 32G 18G 64% /
tmpfs 998M 0 998M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 998M 0 998M 0% /sys/fs/cgroup
tmpfs 200M 0 200M 0% /run/user/0
apt full-upgrade
Preparing to unpack .../libpam-modules-bin_1.4.0-9+deb11u1_amd64.deb ...
dpkg: unrecoverable fatal error, aborting:
unable to flush /var/lib/dpkg/updates/tmp.i after padding: No space left on device
dpkg: error: failed to write status database record about 'apache2' to '/var/lib/dpkg/status': No space left on device
E: Sub-process /usr/bin/dpkg returned an error code (2)
E: Sub-process dpkg --set-selections returned an error code (2)
E: Couldn't revert dpkg selection for approved remove/purge after an error was encountered!
root@universe:~#
reboot
root@universe:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 983M 0 983M 0% /dev
tmpfs 200M 14M 187M 7% /run
/dev/vda1 52G 52G 0 100% /
tmpfs 998M 0 998M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 200M 0 200M 0% /run/user/0
root@universe:~#
apt autoremove
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
dpkg --configure -a
dpkg: error: failed to write status database record about 'apache2' to '/var/lib/dpkg/status': No space left on device
apt autoclean
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.
Personally I like the idea of migrating to a temp server, upgrading the old, and then restoring. By upgrading the old, I mean a fresh install. Since your restores have worked you might skip the migrate and just do a fresh install, install virtualmin from scratch and then pull in the backups.
Still, I think the migrate to a temp server is safer. You can test the setup on the temp to make sure it works. AWS rents space by the hour. Spin up, test, migrate, test some more. Then do a fresh install on the old and spin down the AWS.
You should disable third party repos. You may need to uninstall third party packages, as well, if they present dependency issues or conflicts.
Virtualmin is from a third party repo, too, ours, but I’m confident our packages won’t conflict or cause dependency issues, so you won’t need to, and should not, uninstall any Virtualmin-provided package. But, PHP depends on a bunch of shared libs and likely will present problems during an OS upgrade in the case of third party packages.
A disk space issue seems unlikely with 35GB free when you start it, so you should post the actual error. i.e. what package is being installed when it hits a disk usage error, and the whole exact error.
@Joe - I thought I did post the errors in my previous post. I did not post the entire SSH session in an attempt to keep it short.
QUESTION: Can I update the kernel ONLY without any of the other stuff, and then do the apt full-upgrade?
@ID10T - Yes. Slowly it starts to look like this may be my only option. So far I’ve spend 6+ hours on something that should probably only taken and hour or so.
The bummer is that all this client’s email runs on this server.
If you are in a time crunch, you could outsource the task to a professional like myself. There would be a fee, but also a well laid plan to get the process completed quickly and safely.
Just a point of reference. I didn’t have enough disk space to upgrade discourse. I used up the last of my unallocated LVM space. /dev/mapper/vg00-var 148G 46G 96G 33% /var
It was 100G total and I added the 48G.
`
Is the kernel the problem? If so, your boot partition is what is full, and you’ll need to remove some old kernels before the upgrade. Which is one of the reasons I was asking what package was causing trouble…kernels go to a different place. Your 35GB of free space for / won’t do any good if /boot is full. I don’t see a /boot partition in your df output, though, so I didn’t think that was a good guess.
@Joe - I don’t see a boot partition in my df -h output. This is a virtual server at Vultr so I don’t know if that is the reason why.
@ID10T - Looking at my original (before upgrading) df -h output I don’t think there is enough space remaining anywhere.
@tpnsolutions - Thanks for the offer, but there is no money in this project. It belongs to a private person and the budget is tight. I’ve already got at least 10 hours in this for under a hundred bucks is unless you can do it for $15 we will be flying this one ourselves.
@stefan1959 - Thank you for your reply. The instructions link from Vultr that you sent are nearly the same as the procedure I already used and noted (synopsis) above.
I am awaiting comment from the customer on what they want to do. Either wipe the existing server and fresh re-install the server then restore the websites, users and email. Or spin up another server and copy the new system over to his existing server.
I am kind of leaning on the first, wiping the existing server and re-installing there because I am not sure how the server’s hostname and IP address may change and cause the client email issues when copying the system over from a different server.
