Error! Warning! The 37.25 GiB filesystem mounted at / has no free disk space!

Virtualmin
1
SYSTEM INFORMATION
OS type and version: REQUIRED
Webmin version: REQUIRED
Virtualmin version: REQUIRED
Related products version: RECOMMENDED

Dears,

Could you please be so kind to help me as my website is down with : Error 502 Bad Gateaway. Kindly advise with below log :

  • the reason of the errors
  • how to fix them ( I am new to Webmin/Virtualmin)
  • do I have to fix them using Virtualmin or Webmin ? And how to avoid them in the future ?

When I open the website or the backend ( Wordpress), the browser indicates :

  • Browser working (green)
  • Cloudflare working (green)
  • Host error (red)

I have a E-Commerce website on below server :

1 server CX21 from Hetzner

Installations include:

  • Linux Server - Located in Germany from Hetzner

  • Ubuntu 20.04.3 LTS - End of life April 2030

  • Webmin + Virtualmin

  • Nginx Server - LEMP structure

  • PHP 7.4

  • MySQL

  • phpMyAdmin

I checked in Virtualmin the Nginx Error logs and I found the last 2 lines being errors : :

2021/11/19 15:03:00 [error] 1860469#1860469: *466187 FastCGI sent in stderr: "WordPress database error Error writing file ‘/tmp/MLfd=88’ (OS errno 28 - No space left on device) for query INSERT INTO wp_options (option_name, option_value, autoload) VALUES (‘fa_icons’, 'a:1438:{i:0;a:1:{s:5:“class”;s:11:“fas fa-burn”;}i:1;a:1:{s:5:“class”;s:15:“fas fa-bullseye”;}i:2;a:1:{s:5:“class”;s:15:“fas fa-bullhorn”;}i:3;a:1:{s:5:“class”;s:15:“fas fa-building”;}i:4;a:1:{s:5:“class”;s:15:“far fa-building”;}i:5;a:1:{s:5:“class”;s:10:“fas fa-bug”;}i:6;a:1:{s:5:“class”;s:10:“fab fa-btc”;}i:7;a:1:{s:5:“class”;s:24:“fas fa-briefcase-medical”;}i:8;a:1:{s:5:“class”;s:16:“fas fa-briefcase”;}i:9;a:1:{s:5:“class”;s:14:“fas fa-braille”;}i:10;a:1:{s:5:“class”;s:12:“fas fa-boxes”;}i:11;a:1:{s:5:“class”;s:15:“fas fa-box-open”;}i:12;a:1:{s:5:“class”;s:10:“fas fa-box”;}i:13;a:1:{s:5:“class”;s:19:“fas fa-bowling-ball”;}i:14;a:1:{s:5:“class”;s:15:“fas fa-bookmark”;}i:15;a:1:{s:5:“class”;s:15:“far fa-bookmark”;}i:16;a:1:{s:5:“class”;s:11:“fas fa-book”;}i:17;a:1:{s:5:“class”;s:11:“fas fa-bomb”;}i:18;a:1:{s:5:“class”;s:11:“fas fa-bolt”;}i:19;a:1:{s:5:“class”;s:11:“fas fa-bold”;}i:20;a:1:{s:5:“class”;s:18:“fab fa-bluetooth-b”;}i:21;a:1:{s:5:“class”;s:16:“fab fa-bluetooth”;}i:22;a:1:{s:5:“class”;s:16:“fab fa-blogger-b”;}i:23;a:1:{s:5:“class”;s:14:“fab fa-blogger”;}i:24;a:1:{s:5:“class”;s:12:“fas fa-blind”;}i:25;a:1:{s:5:“class”;s:17:“fab fa-blackberry”;}i:26;a:1:{s:5:“class”;s:16:“fab fa-black-tie”;}i:27;a:1:{s:5:“class”;s:11:“fab fa-bity”;}i:28;a:1:{s:5:“class”;s:14:“fab fa-bitcoin”;}i:29;a:1:{s:5:“class”;s:16:“fab fa-bitbucket”;}i:30;a:1:{s:5:“class”;s:20:“fas fa-birthday-cake”;}i:31;a:1:{s:5:“class”;s:17:“fas fa-binoculars”;}i:32;a:1:{s:5:“class”;s:16:“fab fa-bimobject”;}i:33;a:1:{s:5:“class”;s:14:“fas fa-bicycle”;}i:34;a:1:{s:5:“class”;s:17:“fas fa-bell-slash”;}i:35;a:1:{s:5:“class”;s:17:"far fa-

2021/11/19 15:03:00 [error] 1860469#1860469: *466187 FastCGI sent in stderr: ":“fas fa-arrow-circle-left”;}i:77;a:1:{s:5:“class”;s:24:“fas fa-arrow-circle-down”;}i:78;a:1:{s:5:“class”;s:26:“fas fa-arrow-alt-circle-up”;}i:79;a:1:{s:5:“class”;s:26:“far fa-arrow-alt-circle-up”;}i:80;a:1:{s:5:“class”;s:29:“fas fa-arrow-alt-circle-right”;}i:81;a:1:{s:5:“class”;s:29:“far fa-arrow-alt-circle-right”;}i:82;a:1:{s:5:“class”;s:28:“fas fa-arrow-alt-circle-left”;}i:83;a:1:{s:5:“class”;s:28:“far fa-arrow-alt-circle-left”;}i:84;a:1:{s:5:“class”;s:28:“fas fa-arrow-alt-circle-down”;}i:85;a:1:{s:5:“class”;s:28:“far fa-arrow-alt-circle-down”;}i:86;a:1:{s:5:“class”;s:14:“fas fa-archive”;}i:87;a:1:{s:5:“class”;s:16:“fab fa-apple-pay”;}i:88;a:1:{s:5:“class”;s:12:“fab fa-apple”;}i:89;a:1:{s:5:“class”;s:12:“fab fa-apper”;}i:90;a:1:{s:5:“class”;s:20:“fab fa-app-store-ios”;}i:91;a:1:{s:5:“class”;s:16:“fab fa-app-store”;}i:92;a:1:{s:5:“class”;s:14:“fab fa-angular”;}i:93;a:1:{s:5:“class”;s:20:“fab fa-angrycreative”;}i:94;a:1:{s:5:“class”;s:15:“fas fa-angle-up”;}i:95;a:1:{s:5:“class”;s:18:“fas fa-angle-right”;}i:96;a:1:{s:5:“class”;s:17:“fas fa-angle-left”;}i:97;a:1:{s:5:“class”;s:17:“fas fa-angle-down”;}i:98;a:1:{s:5:“class”;s:22:“fas fa-angle-double-up”;}i:99;a:1:{s:5:“class”;s:25:“fas fa-angle-double-right”;}i:100;a:1:{s:5:“class”;s:24:“fas fa-angle-double-left”;}i:101;a:1:{s:5:“class”;s:24:“fas fa-angle-double-down”;}i:102;a:1:{s:5:“class”;s:16:“fab fa-angellist”;}i:103;a:1:{s:5:“class”;s:14

Thank you so much guys !

2

I don’t know how the error could be clearer? You’re out of disk space. That could mean your disk is literally full, or you assigned a disk quota to your domain that is not big enough.

3

Our website full source code is 2 GIB

Our server disk space is 40 GIB

Where did the othef 38 GIB go , and is it normal to have 38 GIB used when only 1 website is running ?

How can i free up the /dev/sd1 ? As you can see in below command results :
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 385M 944K 384M 1% /run
/dev/sda1 38G 37G 0 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda15 253M 1.1M 252M 1% /boot/efi
tmpfs 385M 0 385M 0% /run/user/0
And how Can I see what inside the /dev/sda ?
Thank you so much in advance

4

This will probably give you clues:

du -xh --max-depth=3 /home

5

At least, I’m assuming it’s in /home. You may have to go up a level to /, which will take longer and be harder to look through.

6

that’s what I found
> du -xh --max-depth=3 /home 8.0K /home/herboristerie-principale/.usermin/mailbox 12K /home/herboristerie-principale/.usermin 4.0K /home/herboristerie-principale/.filemin 12K /home/herboristerie-principale/logs 76K /home/herboristerie-principale/etc/php7.4 80K /home/herboristerie-principale/etc 16K /home/herboristerie-principale/Maildir/.Drafts 16K /home/herboristerie-principale/Maildir/.Sent 4.0K /home/herboristerie-principale/Maildir/cur 16K /home/herboristerie-principale/Maildir/.Trash 8.0K /home/herboristerie-principale/Maildir/new 4.0K /home/herboristerie-principale/Maildir/tmp 72K /home/herboristerie-principale/Maildir 4.0K /home/herboristerie-principale/homes 4.0K /home/herboristerie-principale/virtualmin-backup 44K /home/herboristerie-principale/public_html/.usermin 4.0K /home/herboristerie-principale/public_html/.filemin 48K /home/herboristerie-principale/public_html/.tmp 9.6M /home/herboristerie-principale/public_html/wp-admin 2.7G /home/herboristerie-principale/public_html/wp-content 46M /home/herboristerie-principale/public_html/herbomyadmin 40M /home/herboristerie-principale/public_html/wp-includes 4.0K /home/herboristerie-principale/public_html/.spamassassin 2.8G /home/herboristerie-principale/public_html 796K /home/herboristerie-principale/tmp 4.0K /home/herboristerie-principale/cgi-bin 2.8G /home/herboristerie-principale 2.8G /home
any suggestions?

7

I guess it is bot Home

8

I have the problem in dev/sd1, and the dev/sda1 is outside of Home

9

udev 1946192 0 1946192 0% /dev
tmpfs 393224 944 392280 1% /run
/dev/sda1 39062284 37764832 0 100% /
tmpfs 1966120 0 1966120 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 1966120 0 1966120 0% /sys/fs/cgroup
/dev/sda15 258095 1058 257037 1% /boot/efi
tmpfs 393224 0 393224 0% /run/user/0

10

I want to ask you what the commands to free up the /dev/sda1 ?
Where to go to explore the inside of dev sd 1 and delete folders manually?
+

11

You’re going to need to do some of your own work here. This is basic system administration stuff, you really need to know the basics of how to move around the filesystem and understand what’s going on.

I’ve told you how to find where big files are; it works on any filesystem, not just home (and I told you that in a comment above, suggesting you use du on / as well, if the big files aren’t in home). There is a graphical file manager in Virtualmin/Webmin, it can also show you the size of files and can delete files.

But, you shouldn’t go deleting random files until you understand what the heck is going on. Gigabytes of files shouldn’t just appear out of nowhere. Either you put them there or an attacker did, and you need to find out which. Finding where they are and what they are is the first step to doing that.

1 Like
12

Okay , I’ll do this , thank you I appreciate it . I just want to ask you another questions concerning backups: How to download to local computer the website source code using Virtualmin?

Is it possible to schedule a backup from virtualmin to local computer?

Difference between backup from Webmin and Virtualmin ?

How to download full server contents and configurations using Webmin ?
Because I don’t find the answers in Virtualmin/webmin documentations

13

Like Joe said, you need to find out how those files got there. If you’ve been hacked, you DO NOT want to back that stuff up.

You really need to do one of two things: Figure out what actually happened, fix it entirely, then move on or scrap that drive completely and start over.

And for the record, it’s very easy to do backups in Virtualmin. It’s clearly labeled right there in the main menu.

Virtualmin / Backup and Restore / Backup Virtual Servers

14

thanks , but we cannot do the backup in our PC locally , we can just do it in the server .
I want to do the backup in my own pc .
How Can be possible that we have been hacked , I didn’t get your idea Plz Can you explain?

15

You need to take a few steps back and learn the basics. Your disk is running out of space. Your virtual server container has been assigned XX MB in space, check that. Somebody has been uploading/using your space. Nothing to do with Virtualmin. To backup using Virtualmin Backup and Restoration – Virtualmin Please read the basic FAQs before attempting to manage a server.

Read above, you can backup to your own PC.

16

Any Help !!
when I used this command I have as result:

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 486548 435 486113 1% /dev
tmpfs 491530 750 490780 1% /run
/dev/sda1 2427136 247259 2179877 11% /
tmpfs 491530 4 491526 1% /dev/shm
tmpfs 491530 4 491526 1% /run/lock
tmpfs 491530 18 491512 1% /sys/fs/cgroup
/dev/sda15 0 0 0 - /boot/efi
tmpfs 491530 24 491506 1% /run/user/0
but when I run df-h:
df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 385M 944K 384M 1% /run
/dev/sda1 38G 34G 2.2G 94% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda15 253M 1.1M 252M 1% /boot/efi
tmpfs 385M 0 385M 0% /run/user/0
Any suggestions !!
Thanks

17

Yes, as already said several times - /dev/sda1 38G 34G 2.2G 94% / - that is your problem. I dont know, you or somebody may have uploaded ISO images, porn, public ftp - who knows. You can blast the VPS and start over, its just a VPS after all, and in minutes it will be back online.

18

yes , I see definitely it is not me who uploaded these images , but how can I blast VPS ? like now the website isn’t longer down it backs online , but I want to free up the space of the local disk, and I’m a beginner to manage webservers , so any help form you can help me alot to solve this problem , I want to understand how can I blast The Vps? and this is may not delete the whole website or losing it forever?

19

You can backup your site/ Virtualmin virtual container. Reset VPS and then re import site from backup. You need to do some legwork and search.

20

Okay Thank you !!