Lock files building up in /var/webmin/locks/

robert.minor · October 16, 2024, 8:51pm

SYSTEM INFORMATION
OS type and version	Debian 12
Webmin version	2.202

I am getting lock files in the /var/webmin/locks/####/ folder. Eventually I started getting inode issues with too many of these files in the /var directory. I deleted them and noticed they reappeared. I am not sure why I am getting these continually without any cleanup. I have 2 nearly identical Debian 12 servers that are both getting these. Any ideas, I have done google searches and it doesn’t turn up much in this regard.

lrwxrwxrwx 1 root root 66 Oct 16 15:30 1729110651-3895 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:31 1729110677-3896 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:31 1729110703-3897 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:32 1729110729-3898 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:32 1729110756-3899 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:33 1729110782-3900 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:33 1729110808-3901 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:33 1729110834-3902 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:34 1729110860-3903 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:34 1729110886-3904 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:35 1729110912-3905 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:35 1729110938-3906 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:36 1729110964-3907 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:36 1729110990-3908 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:36 1729111017-3909 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:37 1729111043-3910 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:37 1729111069-3911 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:38 1729111095-3912 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:38 1729111121-3913 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:39 1729111147-3914 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:39 1729111173-3915 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:39 1729111199-3916 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:40 1729111225-3917 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:40 1729111251-3918 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:41 1729111277-3919 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:41 1729111303-3920 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:42 1729111330-3921 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock
lrwxrwxrwx 1 root root 66 Oct 16 15:42 1729111356-3922 → /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock

jimdunn · October 18, 2024, 3:59am

I have a Debian 11 server and have found the same thing… over 48,000+ links in the /var/webmin/locks/446415 folder. I don’t see any inode issues yet, but I hope support can find a way to clean up this folder. They are dated from Oct 3 - 17.

Joe · October 18, 2024, 4:07am

Yeah, I can reproduce this, too.

@Ilia realtime monitoring is leaking lock links.

ls -l 1729076390-62842
lrwxrwxrwx 1 root root 66 Oct 16 10:59 1729076390-62842 -> /var/webmin/modules/authentic-theme/real-time-monitoring.json.lock

That file doesn’t exist, but the link remains. Also, why is this a link?

Ilia · October 18, 2024, 10:00am

Well, initially, I came up with a custom file locking feature specific to the stats server.

However, @Jamie insisted on using the Webmin inbuilt lock_file function.

The reason why those locks are left around and even appear when forcefile flag is used for lock_file is unclear to me at the moment. Perhaps because the default lock_file functions wasn’t meant for endlessly running process…?

Jamie, could you shed light on it? Should I perhaps revert to using my custom locking function? Alternatively, we might not need locking at all, since there’s only one process on the system that writes to that file. It’s sub-optimal, but it would resolve some ongoing locking issues.

Jamie · October 18, 2024, 10:34pm

Those files are created so that it’s possible to find all the open locks Webmin has. However, they should get cleaned up automatically when the lock on the destination file is released … but due to a bug they don’t always

That’s fixed in this patch though : Make sure temporary lock link files are deleted https://forum.virtual… · webmin/webmin@4ef76b2 · GitHub

Ilia · October 18, 2024, 10:36pm

Thanks, Jamie! I will give it a try!

jimdunn · October 19, 2024, 7:33am

I added this to /etc/cron.daily, but I should’ve asked “how many days of locks should be kept”…

#!/bin/sh

locks=/var/webmin/locks

if test -d $locks; then
  /usr/bin/find $locks -ctime +2 -delete > /dev/null
fi

robert.minor · October 19, 2024, 5:10pm

Since this is verified, a fix will get pushed to an update in the near future? I

Ilia · October 19, 2024, 7:37pm

@Jamie, thanks for fixing that bug!

Nevertheless, it didn’t change anything regarding the original problem, where locking is performed from the server. I’m curious… Why would you expect it to work for a process that never finishes? Specifically, I’m referring to this line push(@main::temporary_files, $locklink); of code.

Perhaps I shouldn’t relock the file every time and instead do it just once. This might work, but I don’t think it’s the right solution — it’s more of a workaround. Like what if the stats server gets killed?

Jamie · October 19, 2024, 11:04pm

Maybe the server should lock the file just before writing to it, and unlock afterwards?

Ilia · October 19, 2024, 11:18pm

I think this is exactly what we do currently.

Jamie · October 20, 2024, 12:35am

Can you link to the code?

Ilia · October 20, 2024, 1:12am

Yes, sure! Please have a look:

github.com

webmin/authentic-theme/blob/150d475c09d6c01f446af5fd974311429bab68e1/stats-lib-funcs.pl#L274


      
              my ($graphs_chunk)  = @_;
              # Load stored data
              my $all_stats_histoy = get_stats_history()->{'graphs'};
              # Merge data
              my $graphs = merge_stats($all_stats_histoy, $graphs_chunk);
              # Trim dataset
              trim_stats_history($graphs);
              # Save data
              my $file = "$var_directory/modules/$current_theme".
                              "/real-time-monitoring.json";
              lock_file($file, undef, undef, 1);
              write_file_contents($file, $json->encode($graphs));
              unlock_file($file);
              # Release memory
              undef($graphs_chunk);
              undef($all_stats_histoy);
              undef($graphs);
          }
          
          sub stats_network
          {

Jamie · October 20, 2024, 2:42am

That actually looks fine … however I found another webmin bug where those lock links aren’t removed on unlock. Fixed here : Clean up lock link on unlock https://forum.virtualmin.com/t/lock-file… · webmin/webmin@c1196f7 · GitHub

Ilia · October 20, 2024, 8:45am

Thanks, @Jamie. I see a bug in this new code, though. You’re adding the lock file to %locked_file_link in lock_file, but in unlock_file, you’re trying to clean it up using the real file instead. It won’t work.

Ilia · October 20, 2024, 1:25pm

Furthermore, perhaps related:

Jamie · October 21, 2024, 1:05am

Ok I checked in a further fix for this.

Ilia · October 21, 2024, 8:50am

Thanks, @Jamie! It seems to be working now. Should we also clean up the PID-related directories in /var/webmin/locks since those are accumulating too? Maybe we could just use rmdir for that?

Jamie · October 21, 2024, 11:30pm

Good idea, I’ll also add code for that. Not sure how I missed all these cleanups when I initially implemented this feature…

Ilia · October 22, 2024, 1:00pm

Excellent! Thanks, Jamie!