I run quite a large VPS (80 Gb) and am only running three websites virtually no mail and and openvpn server.
However today when I tried to access my site I could not. Went to webmin and the dashboard told me packages needed updating. However they would not update because my hard disk is full! What with?
I have searched for large files (over 1Gb) but only the virtual file comes up which I have not touched.
I located and deleted a load of archived file (old installation files?) and have got the server back to 94% hard disk usage but it should be about 5%!!!
How do I locate what is happening so I can stop it? Can someone help please?
I’d guess you have something logging pathologically, which would likely be in /var/log somewhere. Though it’s hard to imagine anything that’d log that much.
No, I mean, use du on the path(s) that look interesting.
Obviously the interesting path here is /etc/.git. I can tell you what that path is (it’s the git repo that etckeeper sets up; we install etckeeper as part of the Virtualmin install, just because so many people end up in a situation where they desperately need to recover a config file but aren’t keeping regular backups). But, there is no scenario I can think of where it could grow to 61GB. It should only get new data when something in /etc changes…and the entirety of /etc is only a few megabytes. So, something is changing some file(s) in /etc millions of times in a short time. I can’t think of anything that would do that.
What other software have you installed? Are you doing anything that updates files in /etc extremely often?
You can use git to look at the commit history for clues, maybe?
cd /etc
git log
I don’t know what we’re looking for, as this makes no sense.
Hmmm. I have installed nothing and am not conversant with git development (just read a bit about it) so no idea what is causing it. Here is the output of git log
root@electra:/etc# git log
commit 2d4a3a375ae07f9bf7f401adb70d91bbf076cd1a (HEAD → master)
Author: root root@electra.xsxtc.uk
Date: Sat Feb 11 20:31:31 2023 +0000
daily autocommit
commit 817cb0bd1f1a2801658d2b8cbb92d2ee699dbce3
Author: root root@electra.xsxtc.uk
Date: Sat Feb 11 20:15:01 2023 +0000
Which looks like changes made to the package updates I finally installed tonight? Also, as I said, I deeleted a whole load of files from archives that were all .deb files so I assumed they were old install files which is why I considered it safe to delete them.
Can i safely delete these folders and files in the same way?
Does that output give you an idea of what is going on or do I need to dig into some other logs?G
That’s not where the problem lies. Let’s focus on that /etc/.git dir (which does not contain packages), which is the problem.
And, no. Webmin is not your package manager. Webmin is just calling the system package manager to perform updates. What the package manager does with the packages it downloads is not Webmin’s business. They do get cleaned up, automatically, though. (By the package manager, not Webmin.)
The git log looks normal, and not indicative of anything insane happening (but clearly something insane is happening).
You can’t randomly delete stuff out of /etc/.git, as it’ll just corrupt the git repo and make for lots of errors whenever you do anything that triggers etckeeper to update its repo.
You can do the following, though, which will free up all that space, and, maybe we can try to see it the problem is reproducible.
cd /etc
find . -name ".git" -exec rm -rf {} \;
Be extremely careful with this. It deletes every .git directory and all contents anywhere in the current working dir (and subdirs). Which is what we want…but, if you were to mistype the find command, it’s possible to do a lot of damage.
Then, once the .git dir is gone, reinitialize the repository with:
etckeeper init
This will create a new .git dir, which will hopefully/presumably be tiny. I still can’t think of any reasonable explanation for why it would have 61GB of files.
One thing I could think of…did you, perhaps, use /etc to restore some sort of archive to or something? If a big pile of files appeared in /etc, were committed to git (which happens automatically with etckeeper) and then disappeared again, git would hang on to all the diffs of that…so those files could be recovered. If you accidentally dumped 61GB of files into /etc/ and then deleted them, that would explain this.
No I have not done anything I am aware of. As I say, this is a very simple service to manage a blog as I rebuild the barn we have bought following our house fire, store my old website for my wine business and run a simple OpenVPN server for our use to watch TV here in France. I rarely do anything with the server other than upload media via the wordpress interface. So rarely that I have to remind myself where things are each time I visit!
The websites are set to auto update and I generally have to update packages when I visit webmin but that is it. You will not remember but this happened before (January 2022) and I ended up losing everything and having to move supplier and rebuild from scratch. Cannot say if it was the same problem as OVH destroyed my data but who knows, perhaps it was.
I will continue to monitor the situation to see if it recurs. Is there anything I should be looking at specifically to monitor it?
Thanks for the quick response and excellent guidance Joe.
The /etc/.git/objects folder is at 80M immediately after the reinitialisation. I do not know if this is normal or not. I have set up a reminder to check it daily to see what happens and will respond back here to let you know if it starts growing unusually again.
80M is a little large on a fresh system, though maybe not terrifyingly so. A couple of my production systems are in the 40-50M range, and they’re quite old, in service for years (the .git dir grows forever, but slowly).
You probably should keep an eye on it. That may indicate something is using /etc for storage that should not be (Webmin used to use it for stuff it should not, in the distant past…I guess if you migrated from a very, very, old Virtualmin system from 5+ years ago, it could have that old config that put frequently changing data into /etc).
No migration, this was a completely new build server.
Today du gives 28M for the objects folder. However using the webmin properties function on the same folder gives 40Mib. Not sure why there is a difference?
Well it did until the cat managed to permanently delete the folder from the server by walking on the keyboard!
I assume I can just re-initialise it once more?
OK I re-ran your two commands and it has regenerated the .git folder with contents. The objects folder is now reading 12M. Pesky cat!
I will continue to monitor it. No need to respond thanks.
