Keep previous version of Authentic Theme (to test possible memory leak)

reembar · July 26, 2024, 3:48pm

SYSTEM INFORMATION
OS type and version	Ubuntu Linux 22.04.4 (AWS distribution on Lightsail)
Webmin version	2.201
Authentic Theme version	21.20.3

As I included two subjects in my previous post, I open a new topic to keep things orderly.

When updating from Webmin 2.111 to 2.201 and Virualmin from 7.10.0 to 7.20.2 (and now Usermin from 2.010 to 2.100) the Authetic Theme was updated from 21.10 to 21.20.3

Since refreshing the screen to apply the new theme, I had somewhat higher memory usage and my virtual memory use began to pile up slowly. The top memory user seemed to be /usr/share/webmin/authentic-theme/stats.pl

I stopped my machine when the virtual memory reached 80% and loaded it from a backup (normal to my machin is about 35% for both real memory and virtual memory, after I disabled MySQL and the mail related Clam, Dovecot and Lookup-Domain).

I later did the updates again and let the machine run overnight. When I logged into Webmin this afternoon, the page was stuck on loading (with that strip moving forward and backward on the top) and displayed only the menu on the left. I checked the metrics on my Amazon Lightsail console and found the CPU to be very high since the login. My guess is that it is related to the memory being full. I deleted the machine and loaded it from backup again.

I might have this problem because I have a very weak machine, the 7$ tier in Lightsail that has only 1 GB RAM, 2 vCPUs and 40 GB SSD. But I only run my two private websites so need no more.

I thought I might verify the source of the problem if I keep the 21.10 version of the Authentic Theme while updating to the new Webmin, Virtualmin and Usermin.
I’ll be glad for instructions how to do this, if you find that interesting.

cyberndt · July 26, 2024, 4:25pm

From what I understand Authentic Theme is not to be updated as stand alone.

If you have Virtualmin installed from the Virtualmin installer? You are only to update Webmin/Usermin and Authentic Theme when you receive the message in Package Updates to do so.

Other wise you can break your Virtualmin installation.

Virtualmin version 7.20.2 should have versions
Webmin 2.111
Usermin 2.010
Authentic Theme 21.10

Webmin Standalone install
Webmin 2.201
Authentic Theme 21.20.3

Both these installations on my end show no issue with “memory leak”

Joe · July 26, 2024, 5:49pm

That’s surprising. Normally you’d see a bunch of bigger services; usually, it goes clamd, mariadb, PHP-FPM, Apache, and Webmin and its related processes is somewhere below that (fail2ban looks huge, but it’s mostly mapped, and not actually used).

How are you looking at memory usage? I’d recommend you check in top or htop and sort by memory usage. Keep in mind that RSS/RES is what’s actually in memory, and VIRT is what’s swapped out or memory-mapped, and SHR is memory that can be shared with other processes (so, in the case of Webmin most of Perl and the Perl libraries can be shared across all the processes, as they’re shared libs and very similar…so, if you see multiple Webmin processes consuming few dozen MB, they’re probably sharing at least some of that).

How big is stats.pl? (RES, VIRT, and SHR)

Ilia · July 26, 2024, 7:07pm

I really doubt that stats.pl is the cause of 800MB of RAM usage, considering you have 1G and 80% was full. The expected process size, and what I’ve observed in my vast tests, is around 100-150 MiB max, with an average of 1% CPU load.

Previously, it was significantly worse. I’m confused by this feedback.

If you’re still confident the issue is with the Webmin stats server, could you provide access to the machine so we can run more tests?

reembar · July 27, 2024, 5:28am

Thanks for the replies! I’ll check those memory parameters with htop next time I try this update. Without it my machine went back normal.

When enabling the new authentic theme after the install, my real memory and virtual memory were immediately 10% higher each. When the virtual memory went to 80%, I tried to clear the cache of the theme in it’s interface, but it only added another 10% to the virtual memory.

I used awstats’ interface to look at the memory, but actually stats.pl was only about 118Mib when I stopped the machines and it went back down in one occasion. Below it I had several Apache instances with some 20MiB each that were closed later. I don’t use my machine for databases or receiving email, so disabled clamd and mysql and don’t have mariadb. I do use HTTP/2 but the PHP-FPM was way down and so was fail2ban.

I’ll think a bit about giving access to my machine. There is a saying: “a secret that is known to one is known by no one, a secret that is known to two is known by everyone” but it’s true my website doesn’t contain any state secrets.

reembar · July 31, 2024, 2:14pm

I installed the updates again and checked htop. Indeed It seems the bulk of my virtual memory is made of some 155 “usr/sbin/apache2 -k start” processes. Do the new updates have something to do with the Apache configuration? Like changing from MPM worker to MPM prefork? I have 1G real memory, so I enlarged the virtual memory from 0.5G (which was 35% full before the update and got to 80% after it) to 2G (which is now 23% full) and hope it stabilizes.

Joe · August 1, 2024, 4:26am

Definitely not. We never change your MPM as part of any of our updates.

But, if you installed or upgraded mod_php it would have forced Apache to switch from worker to prefork., and would have made each process a couple hundred MB larger. It is one of many reasons you should never install mod_php.

reembar · August 1, 2024, 7:15am

Thank you! I see on the “Configure Apache Moduls” interface that I’m on mpm_event and checked anyway that I do not have any libapache2-mod-php installed, but will look into it further.

Pallas · August 1, 2024, 10:01pm

Hi Illia, sorry to drop in like this but I can confirm this issue. It’s been happening for the past couple of days and I just chalked it up to something left open on my phone, which sometimes happens with Virtualmin terminal and it baloons in cpu and memory usage until it’s killed.

This time it’s 1727 root 732.37 MiB /usr/bin/perl /usr/share/webmin/authentic-theme/stats.pl 556 and htop confirms Virt 736M Res 732M. This isn’t even the largest RAM usage I saw, a few days ago our server with 4GB of RAM got up to 80% of RAM used. It usually hovers around 40%- the extra 40-ish % was the stats.pl with well over 1GB of RAM used for it. I don’t know the exact numbers as I didn’t think it was important at the time.

This is the second time I see this process top the memory usage. If you’d like I can leave it alone until tomorrow evening (19:00 GMT and later) and let you connect. Keep in mind that this is our production server so if RAM usage rises to the point where it risks OOM-ing something, I’ll have to kill it.

If you’re interested send me a PM or if you have access to it- my personal email.

SYSTEM INFORMATION
OS type and version	Debian Linux 12
Usermin version	2.100
Virtualmin version	7.20.2 Pro
Theme version	21.20.3
Apache version	2.4.61
Package updates	All installed packages are up to date

Ilia · August 1, 2024, 10:41pm

This doesn’t sound right.

However, to be clear, I’ve been running different instances for testing purposes and have never seen such high memory usage by stats.pl. In fact, I have an instance on a super slow Synology with very little memory, and the RAM usage was about 30-40 MB. The other instances are on Parallels, maxing out at 90-100 MB. I also have an instance on GCP and one on a physical machine, and neither has shown such high memory usage.

I’m confused! How much total memory do you have, and what happens if you try to run a memory-intensive task? Does the process get killed, or does stats.pl actually start using less RAM?

Pallas · August 2, 2024, 12:52am

I agree, it doesn’t sound right at all, high ram usage is the exact opposite of what comes to mind when I think of Virtualmin and associated tasks. I’ve been using your software for almost a year now and this is the first time I saw ram ticking up by itself, with no additional tasks, websites or any input.

I’m unable to do ram intensive tests today, but I spun up a cloned server and took screenshots of the initial state(stats.pl is 100mb at startup, ram usage at 23% of 4GB total). By my (extremely) rough estimate based on the uptime of our production server, stats.pl ticks up by 80-100mb ram per day. This behavior happened on another cloned server I spun up for testing a week ago, but sadly I killed it shortly before discovering this thread.

In any case, I’ll do some ram intensive tasks tomorrow evening on our production machine, I don’t want to risk oom killing the db without backing everything up first. Also I’ll keep and eye on the newly spun up server, but it will take some time for stats.pl to grow, I’ll post an update here when and if I see it comfortably surpass 150mb of ram you mentioned.

As for server specs- we’re on entry level Hetzner shared cloud vps: 2vcpu and 4GB ram.

We’re running low traffic wordpress sites at the moment so the only ram intensive tasks I can think of are plugin based migrations, but I don’t think we have websites large enough to fill almost 2GB of available ram. If you have any ideas for better suited ram intensive tasks feel free to share them, I’ll be happy to run them and provide more info.

Aaron_CSNSW · August 2, 2024, 3:23am

Registered to report the same issue. Host is a relatively fresh Ubuntu 22.04.4 instance in Azure, LEMP stack installed via the automated virtualmin-install.sh script. Memory usage of stats.pl grows until the server becomes unresponsive.

Restarting webmin via /usr/bin/webmin restart releases memory, so I’ve implemented this as a nightly cron job for now.

I note there have been some major changes to stats.pl in the authentic-theme repo, it might be worth hobbling along with the nightly webmin restart until these changes make it into the stable apt repos.

SYSTEM INFORMATION
OS type and version	Ubuntu Linux 22.04.4
Usermin version	2.100
Virtualmin version	7.20.2
Theme version	21.20.3
Nginx version	1.18.0

Aaron_CSNSW · August 2, 2024, 3:29am

New reply because I can’t edit the last post to add another image (new user restrictions).

Expanding metrics to 30d shows stable memory usage until Jul 22. Hopefully I can find out what might have changed on the host - will report back if there is anything interesting.

Ilia · August 2, 2024, 10:31am

Sadly, I can now see the same issue on one of my servers, e.g.:

37.2% of 1.9 GB is around 800 MB for the Webmin process. That is quite a lot. And I haven’t seen this amount of RAM used by Webmin except when installing a large number of package updates.

This doesn’t seem right at all. Though I don’t see any processes being killed by the OOM killer. At least not yet.

I will keep investigating this issue. However, I’m not sure right now what could be causing it. But I will dig into it. Also, @Jamie, @Joe, do you have any clues?

Ilia · August 2, 2024, 11:06am

I may have found the issue: we’re not handling connected and disconnected clients properly. I’m running more in-depth tests now …

Pallas · August 2, 2024, 11:46am

I just checked the newly spun up server and my estimate of 100mb per day was conservative by a factor of 2. In 12 hours since the spin up of a new server stats.pl grew from 100 to 200mb. I will reboot the test server, disconnect all clients from it and check in the evening if the stats.pl grew in memory usage.

Also, a server reboot might be unnecessary, it’s sufficient to kill stats.pl and it immediately restarts with initial 100mb usage. As a quick fix, the above mentioned reboot cron could do the job, but a stats.pl restart is sufficient and doesn’t interrupt other processes

Ilia · August 2, 2024, 12:34pm

Yeah, I’ve also been running deeper tests. The sad part is that for me, RAM usage grows to 250-300 MB and then, after about 30 minutes, it drops down to expected ~100 MB. This is very confusing and hard to debug.

Still looking …

Ilia · August 2, 2024, 1:02pm

Pallas:

Ilia:

Pallas:

Also, a server reboot might be unnecessary, it’s sufficient to kill stats.pl and it immediately restarts with initial 100mb usage. As a quick fix, the above mentioned reboot cron could do the job, but a stats.pl restart is sufficient and doesn’t interrupt other processes

Yeah, I’ve also been running deeper tests. The sad part is that for me, RAM usage grows to 250-300 MB and then, after about 30 minutes, it drops down to expected ~100 MB. This is very confusing and hard to debug.

I just checked the newly spun up server and my estimate of 100mb per day was conservative by a factor of 2. In 12 hours since the spin up of a new server stats.pl grew from 100 to 200mb. I will reboot the test server, disconnect all clients from it and check in the evening if the stats.pl grew in memory usage.

I wonder now, do you have JSON::XS Perl module installed on the system where you’re seeing this issue?? If so, can you uninstall it and see if that changes anything?

You can uninstall it on Debian/Ubuntu systems by running:

apt-get remove libjson-xs-perl

or on EL systems:

dnf remove perl-JSON-XS

Ilia · August 2, 2024, 1:23pm

@Jamie, I have tried to look deeper to inspect the memory map of the process on the system where I saw that happening.

What are your thoughts?

So, I tried both pmap -x <pid> and gdb -p <pid>.

`pmap -x <pid> output:`

6313:   /usr/bin/perl /usr/share/webmin/authentic-theme/stats.pl 556
Address           Kbytes     RSS   Dirty Mode  Mapping
0000560654dde000     292       0       0 r---- perl
0000560654e27000    1620    1268       0 r-x-- perl
0000560654fbc000    1732     300       0 r---- perl
000056065516d000      60      60      60 r---- perl
000056065517c000      12      12      12 rw--- perl
000056065517f000      24      24      24 rw---   [ anon ]
0000560655cf0000  770572  770520  770520 rw---   [ anon ]
00007fafd70d9000     516     516     516 rw---   [ anon ]
00007fafd715a000    1028    1028    1028 rw---   [ anon ]
00007fafd7291000     232     232     232 rw---   [ anon ]
00007fafd72dc000     264     260     260 rw---   [ anon ]
00007fafd733f000     132     132     132 rw---   [ anon ]
00007fafd736e000     132     132     132 rw---   [ anon ]
00007fafd73a5000     232     200     200 rw---   [ anon ]
00007fafd73df000       8       0       0 r---- SDBM_File.so
00007fafd73e1000      16       0       0 r-x-- SDBM_File.so
00007fafd73e5000       4       0       0 r---- SDBM_File.so
00007fafd73e6000       4       4       4 r---- SDBM_File.so
00007fafd73e7000       4       4       4 rw--- SDBM_File.so
00007fafd73e8000       8       0       0 r---- Glob.so
00007fafd73ea000      16       4       0 r-x-- Glob.so
00007fafd73ee000       4       0       0 r---- Glob.so
00007fafd73ef000       4       4       4 r---- Glob.so
00007fafd73f0000       4       4       4 rw--- Glob.so
00007fafd73f1000       4       0       0 r---- Socket6.so
00007fafd73f2000      12       0       0 r-x-- Socket6.so
00007fafd73f5000       4       0       0 r---- Socket6.so
00007fafd73f6000       4       4       4 r---- Socket6.so
00007fafd73f7000       4       4       4 rw--- Socket6.so
00007fafd73f8000      24       0       0 r---- POSIX.so
00007fafd73fe000      52       0       0 r-x-- POSIX.so
00007fafd740b000      16       0       0 r---- POSIX.so
00007fafd740f000      12      12      12 r---- POSIX.so
00007fafd7412000       4       4       4 rw--- POSIX.so
00007fafd7413000       4       0       0 r---- attributes.so
00007fafd7414000       4       0       0 r-x-- attributes.so
00007fafd7415000       4       0       0 r---- attributes.so
00007fafd7416000       4       4       4 r---- attributes.so
00007fafd7417000       4       4       4 rw--- attributes.so
00007fafd7418000       8       0       0 r---- XS.so
00007fafd741a000      40      20       0 r-x-- XS.so
00007fafd7424000      12       4       0 r---- XS.so
00007fafd7427000       4       4       4 r---- XS.so
00007fafd7428000       4       4       4 rw--- XS.so
00007fafd7429000       8       0       0 r---- HiRes.so
00007fafd742b000      16       4       0 r-x-- HiRes.so
00007fafd742f000       4       4       0 r---- HiRes.so
00007fafd7430000       4       4       4 r---- HiRes.so
00007fafd7431000       4       4       4 rw--- HiRes.so
00007fafd7432000      12       0       0 r---- Storable.so
00007fafd7435000      84       0       0 r-x-- Storable.so
00007fafd744a000      12       0       0 r---- Storable.so
00007fafd744d000       4       4       4 r---- Storable.so
00007fafd744e000       4       4       4 rw--- Storable.so
00007fafd744f000      12       0       0 r---- Encode.so
00007fafd7452000      24      20       0 r-x-- Encode.so
00007fafd7458000       8       0       0 r---- Encode.so
00007fafd745a000       4       4       4 r---- Encode.so
00007fafd745b000       4       4       4 rw--- Encode.so
00007fafd745c000       4       0       0 r---- SHA.so
00007fafd745d000      32       0       0 r-x-- SHA.so
00007fafd7465000       8       0       0 r---- SHA.so
00007fafd7467000       4       4       4 r---- SHA.so
00007fafd7468000       4       4       4 rw--- SHA.so
00007fafd7469000       4       0       0 r---- Cwd.so
00007fafd746a000       8       0       0 r-x-- Cwd.so
00007fafd746c000       4       0       0 r---- Cwd.so
00007fafd746d000       4       4       4 r---- Cwd.so
00007fafd746e000       4       4       4 rw--- Cwd.so
00007fafd746f000       8       0       0 r---- Fcntl.so
00007fafd7471000       4       0       0 r-x-- Fcntl.so
00007fafd7472000       4       0       0 r---- Fcntl.so
00007fafd7473000       4       4       4 r---- Fcntl.so
00007fafd7474000       4       4       4 rw--- Fcntl.so
00007fafd7475000       4       0       0 r---- Base64.so
00007fafd7476000       8       0       0 r-x-- Base64.so
00007fafd7478000       4       0       0 r---- Base64.so
00007fafd7479000       4       4       4 r---- Base64.so
00007fafd747a000       4       4       4 rw--- Base64.so
00007fafd747b000       4       0       0 r---- MD5.so
00007fafd747c000      12       0       0 r-x-- MD5.so
00007fafd747f000       4       0       0 r---- MD5.so
00007fafd7480000       4       4       4 r---- MD5.so
00007fafd7481000       4       4       4 rw--- MD5.so
00007fafd7482000       8       0       0 r---- Util.so
00007fafd7484000      40      36       0 r-x-- Util.so
00007fafd748e000       8       0       0 r---- Util.so
00007fafd7490000       4       4       4 r---- Util.so
00007fafd7491000       4       4       4 rw--- Util.so
00007fafd7492000      12       0       0 r---- Socket.so
00007fafd7495000      16       0       0 r-x-- Socket.so
00007fafd7499000       8       0       0 r---- Socket.so
00007fafd749b000       8       8       8 r---- Socket.so
00007fafd749d000       4       4       4 rw--- Socket.so
00007fafd749e000      12       8       8 rw---   [ anon ]
00007fafd74a1000       8       0       0 r---- libcrypt.so.1.1.0
00007fafd74a3000      88       0       0 r-x-- libcrypt.so.1.1.0
00007fafd74b9000     104       0       0 r---- libcrypt.so.1.1.0
00007fafd74d3000       4       4       4 r---- libcrypt.so.1.1.0
00007fafd74d4000       4       4       4 rw--- libcrypt.so.1.1.0
00007fafd74d5000      32       0       0 rw---   [ anon ]
00007fafd74dd000     152       4       0 r---- libc.so.6
00007fafd7503000    1364     768       0 r-x-- libc.so.6
00007fafd7658000     332     128       0 r---- libc.so.6
00007fafd76ab000      16      16      16 r---- libc.so.6
00007fafd76af000       8       8       8 rw--- libc.so.6
00007fafd76b1000      52      20      20 rw---   [ anon ]
00007fafd76be000      64      16       0 r---- libm.so.6
00007fafd76ce000     460      76       0 r-x-- libm.so.6
00007fafd7741000     360       4       0 r---- libm.so.6
00007fafd779b000       4       4       4 r---- libm.so.6
00007fafd779c000       4       4       4 rw--- libm.so.6
00007fafd779d000       8       0       0 r---- IO.so
00007fafd779f000       8       0       0 r-x-- IO.so
00007fafd77a1000       4       0       0 r---- IO.so
00007fafd77a2000       4       4       4 r---- IO.so
00007fafd77a3000       4       4       4 rw--- IO.so
00007fafd77a4000       8       8       8 rw---   [ anon ]
00007fafd77a6000       4       0       0 r---- ld-linux-x86-64.so.2
00007fafd77a7000     148     128       0 r-x-- ld-linux-x86-64.so.2
00007fafd77cc000      40      12       0 r---- ld-linux-x86-64.so.2
00007fafd77d6000       8       8       8 r---- ld-linux-x86-64.so.2
00007fafd77d8000       8       8       8 rw--- ld-linux-x86-64.so.2
00007ffe589e5000     132      32      32 rw---   [ stack ]
00007ffe58a8e000      16       0       0 r----   [ anon ]
00007ffe58a92000       8       4       0 r-x--   [ anon ]
---------------- ------- ------- ------- 
total kB          781080  776188  773388

For gdb -p <pid>, I ran it, and then ran info proc mappings followed by dump memory memory_dump.bin 0x560655cf0000 0x560684d99000 but kernel killed the process and all related data wasn’t saved unfortunately:

~# journalctl -k | grep -i 'Out of memory'
Aug 02 16:08:43 cloud-1 kernel: Out of memory: Killed process 6313 (stats.pl) total-vm:781232kB, anon-rss:773464kB, file-rss:8kB, shmem-rss:0kB, UID:0 pgtables:1568kB oom_score_adj:0

`gdb -p 6313 output`:

~# gdb -p 6313
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 6313
Reading symbols from /usr/bin/perl...
(No debugging symbols found in /usr/bin/perl)
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from /usr/lib/debug/.build-id/ba/ac66d4ed827c5168c0c72b9ee3b56ab5750a8a.debug...
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from /usr/lib/debug/.build-id/58/254ca972028402bc40624f81388d85ec95f70d.debug...
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libcrypt.so.1)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/0e/221d13cd73dac3d7b782778a5f325252903c3e.debug...
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Digest/MD5/MD5.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Digest/MD5/MD5.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/MIME/Base64/Base64.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/MIME/Base64/Base64.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Encode/Encode.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Encode/Encode.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Storable/Storable.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Storable/Storable.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Time/HiRes/HiRes.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/Time/HiRes/HiRes.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so)
Reading symbols from /usr/lib/x86_64-linux-gnu/perl/5.36/auto/SDBM_File/SDBM_File.so...
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/perl/5.36/auto/SDBM_File/SDBM_File.so)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffe58a03310, rem=rem@entry=0x7ffe58a03310)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:71
71	../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) info proc mappings
process 6313
Mapped address spaces:

          Start Addr           End Addr       Size     Offset  Perms  objfile
      0x560654dde000     0x560654e27000    0x49000        0x0  r--p   /usr/bin/perl
      0x560654e27000     0x560654fbc000   0x195000    0x49000  r-xp   /usr/bin/perl
      0x560654fbc000     0x56065516d000   0x1b1000   0x1de000  r--p   /usr/bin/perl
      0x56065516d000     0x56065517c000     0xf000   0x38f000  r--p   /usr/bin/perl
      0x56065517c000     0x56065517f000     0x3000   0x39e000  rw-p   /usr/bin/perl
      0x56065517f000     0x560655185000     0x6000        0x0  rw-p   
      0x560655cf0000     0x560684d99000 0x2f0a9000        0x0  rw-p   [heap]
      0x7fafd70d9000     0x7fafd715a000    0x81000        0x0  rw-p   
      0x7fafd715a000     0x7fafd725b000   0x101000        0x0  rw-p   
      0x7fafd7291000     0x7fafd72cb000    0x3a000        0x0  rw-p   
      0x7fafd72dc000     0x7fafd731e000    0x42000        0x0  rw-p   
      0x7fafd733f000     0x7fafd7360000    0x21000        0x0  rw-p   
      0x7fafd736e000     0x7fafd738f000    0x21000        0x0  rw-p   
      0x7fafd73a5000     0x7fafd73df000    0x3a000        0x0  rw-p   
      0x7fafd73df000     0x7fafd73e1000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/SDBM_File/SDBM_File.so
      0x7fafd73e1000     0x7fafd73e5000     0x4000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/SDBM_File/SDBM_File.so
      0x7fafd73e5000     0x7fafd73e6000     0x1000     0x6000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/SDBM_File/SDBM_File.so
      0x7fafd73e6000     0x7fafd73e7000     0x1000     0x6000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/SDBM_File/SDBM_File.so
      0x7fafd73e7000     0x7fafd73e8000     0x1000     0x7000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/SDBM_File/SDBM_File.so
      0x7fafd73e8000     0x7fafd73ea000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so
      0x7fafd73ea000     0x7fafd73ee000     0x4000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so
      0x7fafd73ee000     0x7fafd73ef000     0x1000     0x6000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so
      0x7fafd73ef000     0x7fafd73f0000     0x1000     0x6000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so
      0x7fafd73f0000     0x7fafd73f1000     0x1000     0x7000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob/Glob.so
      0x7fafd73f1000     0x7fafd73f2000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so
      0x7fafd73f2000     0x7fafd73f5000     0x3000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so
      0x7fafd73f5000     0x7fafd73f6000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so
      0x7fafd73f6000     0x7fafd73f7000     0x1000     0x5000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so
      0x7fafd73f7000     0x7fafd73f8000     0x1000     0x6000  rw-p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Socket6/Socket6.so
      0x7fafd73f8000     0x7fafd73fe000     0x6000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so
      0x7fafd73fe000     0x7fafd740b000     0xd000     0x6000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so
      0x7fafd740b000     0x7fafd740f000     0x4000    0x13000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so
      0x7fafd740f000     0x7fafd7412000     0x3000    0x17000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so
      0x7fafd7412000     0x7fafd7413000     0x1000    0x1a000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX/POSIX.so
      0x7fafd7413000     0x7fafd7414000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so
      0x7fafd7414000     0x7fafd7415000     0x1000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so
      0x7fafd7415000     0x7fafd7416000     0x1000     0x2000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so
      0x7fafd7416000     0x7fafd7417000     0x1000     0x2000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so
      0x7fafd7417000     0x7fafd7418000     0x1000     0x3000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/attributes/attributes.so
      0x7fafd7418000     0x7fafd741a000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so
--Type <RET> for more, q to quit, c to continue without paging--
      0x7fafd741a000     0x7fafd7424000     0xa000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so
      0x7fafd7424000     0x7fafd7427000     0x3000     0xc000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so
      0x7fafd7427000     0x7fafd7428000     0x1000     0xe000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so
      0x7fafd7428000     0x7fafd7429000     0x1000     0xf000  rw-p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/JSON/XS/XS.so
      0x7fafd7429000     0x7fafd742b000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Time/HiRes/HiRes.so
      0x7fafd742b000     0x7fafd742f000     0x4000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Time/HiRes/HiRes.so
      0x7fafd742f000     0x7fafd7430000     0x1000     0x6000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Time/HiRes/HiRes.so
      0x7fafd7430000     0x7fafd7431000     0x1000     0x7000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Time/HiRes/HiRes.so
      0x7fafd7431000     0x7fafd7432000     0x1000     0x8000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Time/HiRes/HiRes.so
      0x7fafd7432000     0x7fafd7435000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Storable/Storable.so
      0x7fafd7435000     0x7fafd744a000    0x15000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Storable/Storable.so
      0x7fafd744a000     0x7fafd744d000     0x3000    0x18000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Storable/Storable.so
      0x7fafd744d000     0x7fafd744e000     0x1000    0x1a000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Storable/Storable.so
      0x7fafd744e000     0x7fafd744f000     0x1000    0x1b000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Storable/Storable.so
      0x7fafd744f000     0x7fafd7452000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/Encode.so
      0x7fafd7452000     0x7fafd7458000     0x6000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/Encode.so
      0x7fafd7458000     0x7fafd745a000     0x2000     0x9000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/Encode.so
      0x7fafd745a000     0x7fafd745b000     0x1000     0xa000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/Encode.so
      0x7fafd745b000     0x7fafd745c000     0x1000     0xb000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/Encode.so
      0x7fafd745c000     0x7fafd745d000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so
      0x7fafd745d000     0x7fafd7465000     0x8000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so
      0x7fafd7465000     0x7fafd7467000     0x2000     0x9000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so
      0x7fafd7467000     0x7fafd7468000     0x1000     0xa000  r--p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so
      0x7fafd7468000     0x7fafd7469000     0x1000     0xb000  rw-p   /usr/lib/x86_64-linux-gnu/perl5/5.36/auto/Digest/SHA/SHA.so
      0x7fafd7469000     0x7fafd746a000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so
      0x7fafd746a000     0x7fafd746c000     0x2000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so
      0x7fafd746c000     0x7fafd746d000     0x1000     0x3000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so
      0x7fafd746d000     0x7fafd746e000     0x1000     0x3000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so
      0x7fafd746e000     0x7fafd746f000     0x1000     0x4000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd/Cwd.so
      0x7fafd746f000     0x7fafd7471000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so
      0x7fafd7471000     0x7fafd7472000     0x1000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so
      0x7fafd7472000     0x7fafd7473000     0x1000     0x3000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so
      0x7fafd7473000     0x7fafd7474000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so
      0x7fafd7474000     0x7fafd7475000     0x1000     0x5000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl/Fcntl.so
      0x7fafd7475000     0x7fafd7476000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/MIME/Base64/Base64.so
      0x7fafd7476000     0x7fafd7478000     0x2000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/MIME/Base64/Base64.so
      0x7fafd7478000     0x7fafd7479000     0x1000     0x3000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/MIME/Base64/Base64.so
      0x7fafd7479000     0x7fafd747a000     0x1000     0x3000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/MIME/Base64/Base64.so
      0x7fafd747a000     0x7fafd747b000     0x1000     0x4000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/MIME/Base64/Base64.so
      0x7fafd747b000     0x7fafd747c000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Digest/MD5/MD5.so
      0x7fafd747c000     0x7fafd747f000     0x3000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Digest/MD5/MD5.so
      0x7fafd747f000     0x7fafd7480000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Digest/MD5/MD5.so
      0x7fafd7480000     0x7fafd7481000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Digest/MD5/MD5.so
      0x7fafd7481000     0x7fafd7482000     0x1000     0x5000  rw-p   /usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Digest/MD5/MD5.so
--Type <RET> for more, q to quit, c to continue without paging--
      0x7fafd7482000     0x7fafd7484000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so
      0x7fafd7484000     0x7fafd748e000     0xa000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so
      0x7fafd748e000     0x7fafd7490000     0x2000     0xc000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so
      0x7fafd7490000     0x7fafd7491000     0x1000     0xd000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so
      0x7fafd7491000     0x7fafd7492000     0x1000     0xe000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util/Util.so
      0x7fafd7492000     0x7fafd7495000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so
      0x7fafd7495000     0x7fafd7499000     0x4000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so
      0x7fafd7499000     0x7fafd749b000     0x2000     0x7000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so
      0x7fafd749b000     0x7fafd749d000     0x2000     0x9000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so
      0x7fafd749d000     0x7fafd749e000     0x1000     0xb000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/Socket/Socket.so
      0x7fafd749e000     0x7fafd74a1000     0x3000        0x0  rw-p   
      0x7fafd74a1000     0x7fafd74a3000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
      0x7fafd74a3000     0x7fafd74b9000    0x16000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
      0x7fafd74b9000     0x7fafd74d3000    0x1a000    0x18000  r--p   /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
      0x7fafd74d3000     0x7fafd74d4000     0x1000    0x31000  r--p   /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
      0x7fafd74d4000     0x7fafd74d5000     0x1000    0x32000  rw-p   /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
      0x7fafd74d5000     0x7fafd74dd000     0x8000        0x0  rw-p   
      0x7fafd74dd000     0x7fafd7503000    0x26000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7fafd7503000     0x7fafd7658000   0x155000    0x26000  r-xp   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7fafd7658000     0x7fafd76ab000    0x53000   0x17b000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7fafd76ab000     0x7fafd76af000     0x4000   0x1ce000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7fafd76af000     0x7fafd76b1000     0x2000   0x1d2000  rw-p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7fafd76b1000     0x7fafd76be000     0xd000        0x0  rw-p   
      0x7fafd76be000     0x7fafd76ce000    0x10000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7fafd76ce000     0x7fafd7741000    0x73000    0x10000  r-xp   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7fafd7741000     0x7fafd779b000    0x5a000    0x83000  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7fafd779b000     0x7fafd779c000     0x1000    0xdc000  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7fafd779c000     0x7fafd779d000     0x1000    0xdd000  rw-p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7fafd779d000     0x7fafd779f000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so
      0x7fafd779f000     0x7fafd77a1000     0x2000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so
      0x7fafd77a1000     0x7fafd77a2000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so
      0x7fafd77a2000     0x7fafd77a3000     0x1000     0x4000  r--p   /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so
      0x7fafd77a3000     0x7fafd77a4000     0x1000     0x5000  rw-p   /usr/lib/x86_64-linux-gnu/perl-base/auto/IO/IO.so
      0x7fafd77a4000     0x7fafd77a6000     0x2000        0x0  rw-p   
      0x7fafd77a6000     0x7fafd77a7000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7fafd77a7000     0x7fafd77cc000    0x25000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7fafd77cc000     0x7fafd77d6000     0xa000    0x26000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7fafd77d6000     0x7fafd77d8000     0x2000    0x30000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7fafd77d8000     0x7fafd77da000     0x2000    0x32000  rw-p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffe589e5000     0x7ffe58a06000    0x21000        0x0  rw-p   [stack]
      0x7ffe58a8e000     0x7ffe58a92000     0x4000        0x0  r--p   [vvar]
      0x7ffe58a92000     0x7ffe58a94000     0x2000        0x0  r-xp   [vdso]

(gdb) dump memory memory_dump.bin 0x560655cf0000 0x560684d99000
Cannot access memory at address 0x560663703000

What is the best way to handle it, when I cannot reliably repro this issue?

Ilia · August 2, 2024, 1:28pm

This assumption appears incorrect. I was already handling connections and disconnections correctly.

What’s really frustrating is that all development systems where I can actually run tests behave properly and clear up the memory usage. I can also see it by how the data is handled within the process. So for now, I’m lost!

What I could try as a workaround is disconnecting clients that have been inactive for some time, but so far, that doesn’t seem to make any sense.

Keep previous version of Authentic Theme (to test possible memory leak)

pmap -x <pid> output:

gdb -p 6313 output:

`pmap -x <pid> output:`

`gdb -p 6313 output`: