Thank you very much it Worked.
Still Part 2 I had to add First:
sudo journalctl --rotate
Without, It seemed it didn’t clear anything.
1 Like
I saw Virtualmin 7.20 has been released but could you please confirm if you deployed the fix regarding this issue as I don’t see it in the changelog ?
You should ask @Joe but if it’s not in the changelog … I’m not sure.
Hello, Sorry I reopen the topic because the same problem came back Just today. I didn’t change any setting since my last post. I didn’t try the last Update too.
Still the problem appear when the log seems to take too much place. Which already have been mentioned before.
So in addition of the previous manipulation what I did:
1) I went in /etc/systemd/journald.conf
2) Spotted the #SystemMaxUse= setting which indicate how much space the log can take
3) Replaced by SystemMaxUse=1G
So to anyone facing this problem as you can see in 3) I removed the # (if you don’t it will not apply because it means it’s a comment) and added 1G. Which simply means the log should not take more than 1Go.
So the complete solution (At least I hope) you shall follow if you have the same problem (I retake the answer gave by @Giovxiii )
1) Kill postfix and dovecot:
sudo pkill postfix
sudo pkill dovecot
2) Clear logs:
sudo journalctl --rotate (to archive the log)
sudo journalctl --vacuum-time=1s (to delete was is archived)
3) Mask postfix and dovecot (if it’s only related to the log size, I’m not sure it’s needed):
sudo systemctl mask postfix
sudo systemctl mask dovecot
4) Find the Journal config:
Go in “/etc/systemd/journald.conf”
5) Find:
“#SystemMaxUse=”
6) Replace it by:
“SystemMaxUse=1G”
7) Restart the whole
You do not have to pick 1G it completely depend of your system (Sometime it may even be less)
I think the problem only come from the fact that when you have multiple Site at the same time the Journal scale to it. I don’t know what is the default log size setup by Virtualmin but if you have 3 Websites it sill simply take 3 times the default size (I might be wrong).
Whatever by setting up the max size it should fix your problem. So may be it will give you clues for the next update how to fix it.
Please correct me If I said any mistake.
1 Like
I step back about the Disk Usage possibility because on an second server, with the same setting and ressources, Journalctl take 2.7Go (So more than the first server on which I have problem).
The only difference (according to me) between the first and the second is the First have multiple site (2, but not traffic neither anything complex) while the second only have one.
This is of top of my head.
Is the fact you don’t have lot or ram and a very small swap file might be the issue.
Try increasing your swap to 2gb.
Look at the swapiness value.
Is your system swapping in and out a lot.
1 Like
I have one VM with many more than 3 - one with currently 32!.
(but have not seen this problem) so, is it specific to the type of website? (eg WordPress etc)
CPU load rarely gets above 20% (then only during big updates/mad activity) otherwise remains pretty static (<5%)
1 Like
My journald.conf. Virtualmin with more than 10 WordPress Sites. No problem with memory or CPU (the server has 16 GB RAM):
[Journal]
# Configuración para almacenar los registros de forma persistente
Storage=persistent
# Habilitar la compresión de archivos de registro para ahorrar espacio
Compress=yes
# Sellar los archivos de registro para garantizar la integridad
Seal=yes
# Intervalo de sincronización de los registros en el disco
SyncIntervalSec=5m
# Controlar la tasa de mensajes para evitar inundar los archivos de registro
RateLimitIntervalSec=30s
RateLimitBurst=10000
# Configuración del tamaño máximo de los archivos de registro del sistema
# Limitar el uso total de espacio en disco para los registros del sistema
SystemMaxUse=1G
# Mantener un mínimo de espacio libre en disco
SystemKeepFree=100M
# Tamaño máximo de un solo archivo de registro
SystemMaxFileSize=100M
# Número máximo de archivos de registro
SystemMaxFiles=100
# Configuración del tamaño máximo de los archivos de registro en tiempo de ejecución
# Limitar el uso total de espacio en disco para los registros en tiempo de ejecución
RuntimeMaxUse=500M
# Mantener un mínimo de espacio libre en disco
RuntimeKeepFree=50M
# Tamaño máximo de un solo archivo de registro en tiempo de ejecución
RuntimeMaxFileSize=50M
# Número máximo de archivos de registro en tiempo de ejecución
RuntimeMaxFiles=100
# Tiempo máximo de retención de registros
MaxRetentionSec=1month
# Configuración del tiempo máximo de retención de un archivo de registro individual
MaxFileSec=1month
1 Like
Thank you very much everyone, @shoulders , @Stegan for these fast answers. So lets recap;
- First Server:
2Go Ram, 1Go Virtual Ram, 7Go/30 GO SSD, Debian 12, 1 AMD EPYC 7763, Virtualmin Version: 7.10.0
No traffic (1 Visit per Week), NOT used to send Mail (Even if Postfix is setup), In-house Classic Static Presentation Website.
2 Website on it.
I encountered several time the Problem mentioned above when I added the Second website. And it disappear each time I delete 2 GO of Journal.
- Second Server:
2Go Ram, 1Go Virtual Ram, 10Go/30 GO SSD, Debian 12, 1 AMD EPYC 7543P, Virtualmin Version: 7.8.2
No traffic (1 Visit per Week), Have been used 1 or 2 time to send mails (Only for test), Much more complex platform (But it’s a mock up so …. It’s only for test case)
1 Website on it
Never encountered the Problem
Both have 1GO of Swap (0Mo Used for the first Server, but I clean the Journal Yesterday ….)
As we can see the “SystemMaxUse=1G” it’s 1Go while mine was probably 4GO (the Default limit) before I changed it. So it might also be a clue but you have 16Go of Ram.
Still that It may be Virtualmin which simply do not scale the Max journal Size (“SystemMaxUse”) with the current resource of the system. But Yesterday I putted a limitation of 1GO so I will need to wait 1 or 2 Weeks before to see if it works (The Journal grow average 70Mo per Day).
It would be Great to have feedbacks from @thomas5 @vinc13 @dib @Giovxiii who all 4 Experienced the Problem too. Did it come back or did it completely disappear ? Would be nice to have the current size of Your journal (“journalctl --disk-usage”) ?
I’ve had a similar problem 2 months ago, but the culprit was actually a noisy neighbor.
We’re using shared vcpu and one evening I see journalctl using 100% of cpu and everything slows to a crawl- UI, websites, tar backups… I check every log I could find- nothing, dmesg throws the usual stuff, absolutely no trace of anything going wrong anywhere, it was just slow- that’s it.
I reboot the server and it takes forever to complete, and when it does it’s still on 100% cpu constantly. So I decide to take a snapshot and spin up a new server to troubleshoot the issue while the production server slowly but surely powers on. New server spins up and gives me cpu 1% at idle. I realized that the issue was with the vps parent, not our vps or virtualmin.
I went back to the production server and moved it to a different vps parent by scaling it up and immediately back down- problem solved, 1% cpu at idle ever since with occassional blips but nothing permanent.
I’m not sure if it applies here, but it solved our issue with (apparent) journalctl cpu hogging session- the culprit was not even in our system.
For this particular problem, when collecting mail logs using journalctl, the process execution can take a long time and significant load spike on low-grade CPUs or systems with very little shared resources. This is expected (with low CPU specs) especially after a reboot. Webmin, along with other processes, will run various tasks, including status collection with commands like journalctl -u postfix* -u dovecot*, which can take a long time and somewhat appear like a stuck process.
I have witnessed this on an old and low-performant Intel Celeron CPU. I even wondered what was going on. Running journalctl -u postfix* -u dovecot* in a separate process hanged for minutes until all post-reboot processes, including Webmin’s, are finished. In application to Webmin specifically, those are displayed under the Webmin CGroup in systemctl status webmin command output.
Later, on the same system with low CPU specs, running journalctl -u postfix* -u dovecot* again completed in 2-3 seconds. I could hear the system struggling because this low-grade virtual machine is on a Synology NAS with old spinning HDDs.
On slightly more performant CPUs, with SSDs, this won’t even be an issue. “Slightly” is the keyword here!
So anyone experiencing this issue either needs to share more resources or live with it. There is not much we can do.
To be clear, I have a number of low-cost systems, like $5-10 a month, which also have very limited resources compared to a bare metal machine but still work fairly well.
I’m not sure to follow you. You said the problem was a neighbour but then to solve the issue you indicate:
If the problem was solved by scaling up it have nothing about with the neighbour. As we all said before it might be the available resources. But @dib said he have 8Gb RAM so it would have been nice to have the Log max size too.
And don’t forget each time you setup a new VPS the whole Journal size fall to 0MO.
How did you deduced it was the Neighbour and what was the spec of each VPS ?
But It Might be easily fixed by Virtualmin by decreasing the Default Log max size and make it match the available resources (Let’s say maximum half of it).
Still some clues are missing.
Hello guys,
The problem completely disappeared since the manipulation. FYI my server is 2vcore AMD Epyc 2.6Ghz, 2gb of ram, 2gb of swap and 40Gb SSD.
I started to have this problem when I did an update of Virtualmin in April.
Again, I’m not an expert but @Ilia wrote something about a “bug… fixed just a couple of weeks ago” and about a “small Virtualmin update” that I haven’t seen in the update logs. So I don’t understand why we’re now talking about server performance as a potential cause of the problem.
During 4 months (between December to April), I was able to run 7 low trafic websites (6 wordpress + 1 NextCloud) on that same server. After the update, the CPU went wild as you can see on the graph I posted on May 15th !
Since the manipulation published on May 23, my cpu went back to normal and I added one more wordpress website.
In June, one of my websites for a football competition got a user spike during 6 days of about 800 daily users and more than 5k pages viewed per day. The server held the load like a charm and I was able to publish live score updates.
I’m sorry guys but I really don’t understand why on earth we’re talking about resources as the main problem.
It was not so much global performance but the ram and swap (which can be increased easily). Then I don’t know if it have been fixed or not in the last version.
You indicate 2Gb of swap did the problem happen when it was full ? And how much “journalctl --disk-usage” give currently ? Me I’m at 700Mo currently. But before the fix mentioned above the swap might have been full.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.