@Jamie, another thought! What happens to the Webmin cache is such case scenario? I mean, the stats server internally uses quite a lot of the Webmin APIs, which in turn use different functions that may involve caching.
Could that be the culprit of the problem, you think?
I think I have nailed it!
Please apply this patch and let me know if it fixes the problem for you:
webmin patch https://github.com/webmin/authentic-theme/commit/cdb0c9c
That’s true, but the cache should be a fixed size and not continue to grow unbounded…
Do you have any idea what else could cause the size of stats.pl to grow? I have checked everything multiple times, and I could only find one potential issue, which I believe I fixed in the patch above.
Maybe one work-around would be to have it terminate and restart every hour, and force clients to re-connect?
This would be a work-around. I think I have found more flaws. Fixing it now …
@Jamie, I have dug in more and realized that calling the Webmin API every second is costly and unnecessary, as well as calling eval "use JSON::PP"; on every call to encode/decode JSON. Additionally, it was inconsistent when JSON::XS was installed, as we should have used JSON::XS in that case.
All of these issues are fixed now.
Could you apply this other patch and give it a try?
webmin patch https://github.com/webmin/authentic-theme/commit/6fdd7e9
Let me know if it resolves the issue!
At last, please apply this patch as well:
webmin patch https://github.com/webmin/authentic-theme/commit/7293ee
I truly hope this solves the initial problem! The memory usage may grow slightly after the initial Webmin start, but only by a tiny bit.
Deployed on test server, mem usage for stats.pl immediately went from 200 to 92mb used. I’m going to leave it like this until morning (GMT) and report back if it’s ok.
Sorry for no responses, my friday took several unexpected turns.
I was very happy to read about your discoveries regarding webmin api and json encode/decode. I’m glad this unfortunate and obscure bug produced a robust patch that even improved efficiency and consistency. Thank you
Actually, thank you! But let’s first see how it goes in practice.
Looking forward …
It seems to me that with the last patch (and also when trying the first patch before) the CPU dial is not moving: staying on 0 when logging in and on 80% or 90% after reboot or a shutdown. There also seem to be a considerable delay in drawing and updating the Stats History graphs, that sometimes are not drawn at all, but that seem to be a change that came with the 21.20.3 version.
This is expected, taking about 3-5 seconds on the first connection before graphs start moving.
The CPU load used by stats.pl averages 0.7%, sometimes dropping to 0.3% and occasionally reaching 2%. 85% load is impossible!
However, sadly, it didn’t resolve the original issue! I still see a very slow but incremental memory growth over time. I will keep looking!
What i mean is merely the display: the CPU dial on top of the webmin page seems not to move at all, being stuck on a certain value. The Stats History graphics also sometimes never appear for me, even if I wait several minutes, and if it does, it stops moving frequently for several seconds. I think I noticed those things only after logging out and in of webmin or rebooting.
The stats are supposed to stop for a second or two during collection time. It purely depends on your overall system’s performance. What are your system’s specs?
1GB RAM, 2GB SWAP, 2 vCPUs and 40 GB SSD. I only serve two sites with very low activity, so the resources I use are very low.
When I refresh the display using that round arrow on the right, the dials are moving for a moment, but then stop moving altogether. The graphs also does not appear. I think it came about only after logging out of webmin and in again.
None of this is clear. Can you instead show top and free -h commands output?
Just checked the test server and indeed the memory usage issue is still there.
I’m getting dials and graphs, but I notice the dials are no longer moving smoothly all the time. Cpu dial starts filling up smoothly as usual but shortly after starts jumping around from one value to the next until settling on 0-1% at idle. The erratic behavior doesn’t show abnormal usage, it’s all perfectly within what I see every time I open up virtualmin as it fetches the stats, but the display of values begins stuttering after a while.
Our test server is available for you if you need/want it to troubleshoot the issue.
Sadly, yes. I’m still working on trying to figure out what’s wrong.
Yes, this is normal.
Thanks, but no need! I can clearly see the issue.
All good here!
