Webalizer update doesn't seem to be working correctly

Virtualmin
1

I’m not sure if this is a bug or what. Every day LFD reports to me that the webalizer update is using up too many processes for one of my users (my_user), so this is what caused me to notice that there might be a problem. Here is the report:

Time: Wed Mar 20 05:35:33 2013 -0400 Account: my_user Process Count: 13 (Not killed)

Process Information:

User:my_user PID:14297 PPID:14296 Run Time:31(secs) Memory:37448(kb) exe:/bin/su cmd:su my_user -c /usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o \/home\/my_user\/public_html\/stats -p \/var\/log\/virtualmin\/my\-website\.com_access_log\.5\.gz
User:my_user PID:14298 PPID:14297 Run Time:31(secs) Memory:3956(kb) exe:/bin/dash cmd:sh -c /usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o \/home\/my_user\/public_html\/stats -p \/var\/log\/virtualmin\/my\-website\.com_access_log\.5\.gz
User:my_user PID:14299 PPID:14298 Run Time:31(secs) Memory:50180(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14336 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14337 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14338 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14339 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14340 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14341 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14342 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14343 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14344 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz
User:my_user PID:14345 PPID:14299 Run Time:29(secs) Memory:56488(kb) exe:/usr/bin/webalizer cmd:/usr/bin/webalizer -c /etc/webmin/webalizer/var_log_virtualmin_my-website.com_access_log.conf -o /home/my_user/public_html/stats -p /var/log/virtualmin/my-website.com_access_log.5.gz

It is weird for several reasons:

  • It seems to be executing exactly the same command multiple times. Presumably it should only need to run it once per log file
  • Executing the command /etc/webmin/webalizer/webalizer.pl /var/log/virtualmin/my-website.com_access_log manually from the command time is EXTREMELY slow. My website gets around 500-700 unique visitors per day, but it takes the command around 8 minutes to run.
  • All of the website access logs are time-stamped, however to update, it seems like it parses ALL of the log files. If I execute the above command and then immediately after it finishes, I execute it again, it still takes the same ~8 minutes to execute the second time. It should only need to parse the logs up until where the last update left off. This is what AWStats does and to update the stats a second time takes only a couple of seconds.

Is this a bug with Webmin, or with Webalizer, or not a bug at all?

2

Hmm. Looks like I found out how to speed it up. In Webmin > Servers > Webalizer Logfile Analysis > “my-website.com_access_log” > Edit Options Under the option DNS lookup processes it was set to 10. Changing this to None causes the update to only take 1-2 seconds. This would also explain why there were so many processes parsing the same log file.

However, since it takes so long doing consecutive updates with DNS lookups enabled, perhaps it is not correctly caching the DNS queries and is also still parsing ALL of the log files.

My “Webalizer DNS Cache File” is set to dns_cache.db.