My Virtualmin GPL server recently completely freaked out while attempting to access some of its Web pages I received 500 Internal Server Errors. After ruling out network issues, it appears the problem was that the server’s CPU and Memory usage were so crazy that it would no longer respond to any network connections even ssh log in attempts would TCP timeout.
I created a ticket with my hosting company, and the sysadmin on their end was able to ssh into the server using the hosting provider’s private network. There he determined that the problem was tons of clamscan process all trying to run at the same time. Check out the attached graph from their monitoring software. Max system load was 94.8! How could there possibly be 94.8 runnable processes at the same time!!!
I had no idea that clamscan was even running on the system. I didn’t think the server checked SPAMs for virusus, but apparently it tries to.
The hosting sysadmin disabled clamscan by setting its permissions of 000. Now there are Permission denied errors in procmail’s log file.
I have no clue how to debug this mess. Please help me any way you can. Also, could someone knowledgeable please explain Virtualmin GPL’s mail system, and how and where clamscan fits into it.
I can’t debug this problem, because I don’t know enough information about it yet.
Thanks,
Dave.
Below are some snippets from some log files. I can reply attaching bigger chunks if needed, but I’d rather not due to leaking domain names and IP address and email address, and all of that crap.
procmail.log:
From oGKVfbY575@ahahe.com Wed Jul 3 03:28:52 2013 Subject: [SPAM] The 50 Best Foods for Weight Loss Folder: /home/someuser/Maildir/.spam/new/1372840164.5970_0.somedom 14946 Time:1372840164 From:oGKVfbY575@ahahe.com To:d2e711c7@anotherdomain.com User:someuser Size:14998 Dest:/home/mybadmin/Maildir/.spam/new/1372840164.5970_0.mydomain.com Mode:Spam sh: /usr/bin/clamscan: Permission denied
messages (Around the time of the insane loadavg, it’s full of OOM Killer logs.)
Jul 3 02:18:29 web4 kernel: controller invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Jul 3 02:18:30 web4 kernel: controller cpuset=/ mems_allowed=0 Jul 3 02:18:44 web4 kernel: Pid: 8758, comm: controller Not tainted 2.6.32-358.11.1.el6.x86_64 #1 Jul 3 02:18:44 web4 kernel: Call Trace: Jul 3 02:18:44 web4 kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 Jul 3 02:18:44 web4 kernel: [] ? dump_header+0x90/0x1b0 Jul 3 02:18:44 web4 kernel: [] ? __delayacct_freepages_end+0x2e/0x30 Jul 3 02:18:44 web4 kernel: [] ? security_real_capable_noaudit+0x3c/0x70 Jul 3 02:18:44 web4 kernel: [] ? oom_kill_process+0x82/0x2a0 Jul 3 02:18:44 web4 kernel: [] ? select_bad_process+0xe1/0x120 Jul 3 02:18:44 web4 kernel: [] ? out_of_memory+0x220/0x3c0 Jul 3 02:18:44 web4 kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0 Jul 3 02:18:44 web4 kernel: [] ? alloc_pages_current+0xaa/0x110 Jul 3 02:18:44 web4 kernel: [] ? __page_cache_alloc+0x87/0x90 Jul 3 02:18:44 web4 kernel: [] ? find_get_page+0x1e/0xa0 Jul 3 02:18:44 web4 kernel: [] ? filemap_fault+0x1a7/0x500 Jul 3 02:18:44 web4 kernel: [] ? __do_fault+0x54/0x530 Jul 3 02:18:44 web4 kernel: [] ? handle_pte_fault+0xf7/0xb50 Jul 3 02:18:44 web4 kernel: [] ? __ip_local_out+0x9f/0xb0 Jul 3 02:18:44 web4 kernel: [] ? ip_local_out+0x25/0x30 Jul 3 02:18:44 web4 kernel: [] ? ip_queue_xmit+0x190/0x420 Jul 3 02:18:44 web4 kernel: [] ? copy_user_generic+0xe/0x20 Jul 3 02:18:44 web4 kernel: [] ? handle_mm_fault+0x23a/0x310 Jul 3 02:18:44 web4 kernel: [] ? __do_page_fault+0x139/0x480 Jul 3 02:18:44 web4 kernel: [] ? wait_consider_task+0x9d/0xb20 Jul 3 02:18:44 web4 kernel: [] ? read_tsc+0x9/0x20 Jul 3 02:18:44 web4 kernel: [] ? ktime_get_ts+0xb1/0xf0 Jul 3 02:18:44 web4 kernel: [] ? poll_select_copy_remaining+0xf8/0x150 Jul 3 02:18:44 web4 kernel: [] ? do_page_fault+0x3e/0xa0 Jul 3 02:18:44 web4 kernel: [] ? page_fault+0x25/0x30 Jul 3 02:18:44 web4 kernel: Mem-Info: Jul 3 02:18:44 web4 kernel: Node 0 DMA per-cpu: Jul 3 02:18:44 web4 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 2: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 3: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 4: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 5: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 6: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 7: hi: 0, btch: 1 usd: 0 Jul 3 02:18:44 web4 kernel: Node 0 DMA32 per-cpu: Jul 3 02:18:44 web4 kernel: CPU 0: hi: 186, btch: 31 usd: 34 Jul 3 02:18:44 web4 kernel: CPU 1: hi: 186, btch: 31 usd: 30 Jul 3 02:18:44 web4 kernel: CPU 2: hi: 186, btch: 31 usd: 5 Jul 3 02:18:44 web4 kernel: CPU 3: hi: 186, btch: 31 usd: 3 Jul 3 02:18:44 web4 kernel: CPU 4: hi: 186, btch: 31 usd: 9 Jul 3 02:18:44 web4 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Jul 3 02:18:44 web4 kernel: CPU 7: hi: 186, btch: 31 usd: 2 Jul 3 02:18:44 web4 kernel: active_anon:310818 inactive_anon:104950 isolated_anon:1120 Jul 3 02:18:44 web4 kernel: active_file:363 inactive_file:568 isolated_file:0 Jul 3 02:18:44 web4 kernel: unevictable:1 dirty:3 writeback:176 unstable:0 Jul 3 02:18:44 web4 kernel: free:13204 slab_reclaimable:3226 slab_unreclaimable:13473 Jul 3 02:18:44 web4 kernel: mapped:327 shmem:26 pagetables:18014 bounce:0 Jul 3 02:18:44 web4 kernel: Node 0 DMA free:8264kB min:336kB low:420kB high:504kB active_anon:1372kB inactive_anon:5676kB active_file:12kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15268kB mlocked:0kB dirty:0kB writeback:4kB mapped:12kB shmem:4kB slab_reclaimable:28kB slab_unreclaimable:124kB kernel_stack:0kB pagetables:92kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:218 all_unreclaimable? no Jul 3 02:18:44 web4 kernel: lowmem_reserve[]: 0 1982 1982 1982 Jul 3 02:18:44 web4 kernel: Node 0 DMA32 free:44856kB min:44716kB low:55892kB high:67072kB active_anon:1241900kB inactive_anon:413868kB active_file:1440kB inactive_file:2212kB unevictable:4kB isolated(anon):4480kB isolated(file):0kB present:2030100kB mlocked:4kB dirty:12kB writeback:700kB mapped:1296kB shmem:100kB slab_reclaimable:12876kB slab_unreclaimable:53768kB kernel_stack:3696kB pagetables:71964kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 3 02:18:44 web4 kernel: lowmem_reserve[]: 0 0 0 0 Jul 3 02:18:44 web4 kernel: Node 0 DMA: 6*4kB 10*8kB 16*16kB 5*32kB 3*64kB 3*128kB 2*256kB 3*512kB 1*1024kB 2*2048kB 0*4096kB = 8264kB Jul 3 02:18:44 web4 kernel: Node 0 DMA32: 1528*4kB 840*8kB 628*16kB 324*32kB 67*64kB 14*128kB 8*256kB 5*512kB 1*1024kB 0*2048kB 0*4096kB = 44960kB Jul 3 02:18:44 web4 kernel: 25022 total pagecache pages Jul 3 02:18:44 web4 kernel: 24007 pages in swap cache Jul 3 02:18:44 web4 kernel: Swap cache stats: add 4307521, delete 4283514, find 142406403/142858653 Jul 3 02:18:44 web4 kernel: Free swap = 4kB Jul 3 02:18:44 web4 kernel: Total swap = 1048568kB Jul 3 02:18:44 web4 kernel: 522224 pages RAM Jul 3 02:18:44 web4 kernel: 47365 pages reserved Jul 3 02:18:44 web4 kernel: 31665 pages shared Jul 3 02:18:44 web4 kernel: 453364 pages non-shared Jul 3 02:18:44 web4 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jul 3 02:18:44 web4 kernel: [ 556] 0 556 2716 1 0 -17 -1000 udevd Jul 3 02:18:44 web4 kernel: [ 2401] 0 2401 2660 1 0 -17 -1000 udevd Jul 3 02:18:44 web4 kernel: [ 2402] 0 2402 2715 1 1 -17 -1000 udevd Jul 3 02:18:44 web4 kernel: [ 2583] 0 2583 1539 2 0 0 0 portreserve Jul 3 02:18:44 web4 kernel: [ 2590] 0 2590 62367 182 3 0 0 rsyslogd Jul 3 02:18:44 web4 kernel: [ 2644] 0 2644 2707 94 0 0 0 irqbalance Jul 3 02:18:44 web4 kernel: [ 8228] 81 8228 5383 2 0 0 0 dbus-daemon Jul 3 02:18:44 web4 kernel: [ 8257] 0 8257 1019 1 1 0 0 acpid Jul 3 02:18:44 web4 kernel: [ 8266] 68 8266 6340 136 1 0 0 hald Jul 3 02:18:44 web4 kernel: [ 8267] 0 8267 4526 2 0 0 0 hald-runner Jul 3 02:18:44 web4 kernel: [ 8295] 0 8295 5055 2 1 0 0 hald-addon-inpu Jul 3 02:18:44 web4 kernel: [ 8306] 68 8306 4451 2 0 0 0 hald-addon-acpi Jul 3 02:18:44 web4 kernel: [ 8323] 0 8323 16029 1 0 -17 -1000 sshd Jul 3 02:18:44 web4 kernel: [ 8331] 38 8331 7540 74 0 0 0 ntpd Jul 3 02:18:44 web4 kernel: [ 8367] 0 8367 27050 2 2 0 0 mysqld_safe Jul 3 02:18:44 web4 kernel: [ 8543] 0 8543 4814 2 5 0 0 dovecot Jul 3 02:18:44 web4 kernel: [ 8544] 97 8544 3243 2 1 0 0 anvil Jul 3 02:18:44 web4 kernel: [ 8546] 0 8546 3276 2 3 0 0 log Jul 3 02:18:45 web4 kernel: [ 8555] 0 8555 16602 2 1 0 0 saslauthd Jul 3 02:18:45 web4 kernel: [ 8556] 0 8556 16602 2 0 0 0 saslauthd Jul 3 02:18:45 web4 kernel: [ 8557] 0 8557 16602 2 4 0 0 saslauthd Jul 3 02:18:45 web4 kernel: [ 8558] 0 8558 16602 2 0 0 0 saslauthd Jul 3 02:18:45 web4 kernel: [ 8559] 0 8559 16602 2 1 0 0 saslauthd Jul 3 02:18:45 web4 kernel: [ 8635] 0 8635 19682 57 1 0 0 master Jul 3 02:18:45 web4 kernel: [ 8642] 89 8642 19816 170 0 0 0 qmgr Jul 3 02:18:45 web4 kernel: [ 8663] 0 8663 37546 41 1 0 0 proftpd Jul 3 02:18:45 web4 kernel: [ 8685] 0 8685 27543 2 0 0 0 abrtd Jul 3 02:18:45 web4 kernel: [ 8693] 0 8693 27015 47 1 0 0 abrt-dump-oops Jul 3 02:18:45 web4 kernel: [ 8701] 0 8701 85462 280 6 0 0 httpd Jul 3 02:18:45 web4 kernel: [ 8710] 0 8710 29308 57 4 0 0 crond Jul 3 02:18:45 web4 kernel: [ 8721] 0 8721 5363 1 1 0 0 atd Jul 3 02:18:45 web4 kernel: [ 8731] 0 8731 25230 7 1 0 0 rhnsd Jul 3 02:18:45 web4 kernel: [ 8739] 0 8739 25971 2 0 0 0 rhsmcertd Jul 3 02:18:45 web4 kernel: [ 8752] 0 8752 1604 12 1 0 0 nimbus Jul 3 02:18:45 web4 kernel: [ 8758] 0 8758 2228 88 5 0 0 controller Jul 3 02:18:45 web4 kernel: [ 8773] 89 8773 19701 72 1 0 0 tlsmgr Jul 3 02:18:45 web4 kernel: [ 8776] 0 8776 21085 102 1 0 0 spooler Jul 3 02:18:45 web4 kernel: [ 8780] 0 8780 2251 67 4 0 0 hdb Jul 3 02:18:45 web4 kernel: [ 8791] 0 8791 3076 113 4 0 0 cdm Jul 3 02:18:45 web4 kernel: [ 8812] 0 8812 2982 76 2 0 0 processes Jul 3 02:18:45 web4 kernel: [ 8820] 0 8820 23375 134 1 0 0 miniserv.pl Jul 3 02:18:45 web4 kernel: [ 8839] 0 8839 23500 296 0 0 0 miniserv.pl Jul 3 02:18:45 web4 kernel: [ 8850] 0 8850 1015 2 0 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8852] 0 8852 1015 2 3 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8854] 0 8854 1015 2 7 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8856] 0 8856 1015 2 3 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8858] 0 8858 1015 2 6 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8860] 0 8860 1015 2 0 0 0 mingetty Jul 3 02:18:45 web4 kernel: [ 8863] 0 8863 1019 2 0 0 0 agetty Jul 3 02:18:45 web4 kernel: [ 9127] 0 9127 23299 58 0 -17 -1000 auditd Jul 3 02:18:45 web4 kernel: [ 3094] 0 3094 48304 386 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [15547] 542 15547 55052 16 0 0 0 php-cgi Jul 3 02:18:45 web4 kernel: [19869] 549 19869 54951 2 0 0 0 php-cgi Jul 3 02:18:45 web4 kernel: [20597] 557 20597 54924 2 4 0 0 php-cgi Jul 3 02:18:45 web4 kernel: [ 7082] 0 7082 9814 2 2 0 0 ssl-params Jul 3 02:18:45 web4 kernel: [16296] 0 16296 35029 6 2 0 0 crond Jul 3 02:18:45 web4 kernel: [16298] 0 16298 39793 2 0 0 0 backup.pl Jul 3 02:18:45 web4 kernel: [16330] 0 16330 38845 756 0 0 0 lfd Jul 3 02:18:45 web4 kernel: [19366] 0 19366 19777 15 0 0 0 local Jul 3 02:18:45 web4 kernel: [19463] 0 19463 2307 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [19464] 0 19464 2307 1 4 0 0 sh Jul 3 02:18:45 web4 kernel: [19465] 0 19465 7010 58 0 0 0 tar Jul 3 02:18:45 web4 kernel: [19466] 0 19466 1074 75 4 0 0 gzip Jul 3 02:18:45 web4 kernel: [19467] 0 19467 1024 10 0 0 0 cat Jul 3 02:18:45 web4 kernel: [20085] 0 20085 19777 2 0 0 0 local Jul 3 02:18:45 web4 kernel: [21402] 0 21402 19777 18 0 0 0 local Jul 3 02:18:45 web4 kernel: [21591] 0 21591 28736 895 1 0 0 miniserv.pl Jul 3 02:18:45 web4 kernel: [21775] 0 21775 36300 30 1 0 0 miniserv.pl Jul 3 02:18:45 web4 kernel: [21816] 0 21816 2833 2 0 0 0 sh Jul 3 02:18:45 web4 kernel: [21822] 0 21822 18905 250 0 0 0 rpm Jul 3 02:18:45 web4 kernel: [21901] 0 21901 39557 229 0 0 0 miniserv.pl Jul 3 02:18:45 web4 kernel: [21923] 0 21923 19777 115 0 0 0 local Jul 3 02:18:45 web4 kernel: [22057] 0 22057 2833 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [22060] 0 22060 18862 748 4 0 0 rpm Jul 3 02:18:45 web4 kernel: [22256] 557 22256 2307 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [22266] 557 22266 41732 11725 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [22383] 0 22383 19777 114 0 0 0 local Jul 3 02:18:45 web4 kernel: [22414] 48 22414 85495 362 2 0 0 httpd Jul 3 02:18:45 web4 kernel: [22660] 0 22660 19777 11 0 0 0 local Jul 3 02:18:45 web4 kernel: [22725] 0 22725 19777 115 1 0 0 local Jul 3 02:18:45 web4 kernel: [22980] 0 22980 19777 27 4 0 0 local Jul 3 02:18:45 web4 kernel: [23014] 557 23014 2307 2 4 0 0 sh Jul 3 02:18:45 web4 kernel: [23015] 557 23015 57619 13078 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [23237] 0 23237 19777 20 0 0 0 local Jul 3 02:18:45 web4 kernel: [23386] 0 23386 19777 114 0 0 0 local Jul 3 02:18:45 web4 kernel: [23429] 89 23429 19702 85 5 0 0 pickup Jul 3 02:18:45 web4 kernel: [23605] 0 23605 19777 21 0 0 0 local Jul 3 02:18:45 web4 kernel: [23947] 0 23947 38996 440 5 0 0 lfd Jul 3 02:18:45 web4 kernel: [24014] 48 24014 85495 359 3 0 0 httpd Jul 3 02:18:45 web4 kernel: [24081] 48 24081 85495 387 2 0 0 httpd Jul 3 02:18:45 web4 kernel: [24082] 0 24082 35029 6 2 0 0 crond Jul 3 02:18:45 web4 kernel: [24137] 0 24137 34458 5826 0 0 0 monitor.pl Jul 3 02:18:45 web4 kernel: [24200] 0 24200 19777 17 0 0 0 local Jul 3 02:18:45 web4 kernel: [24267] 48 24267 85499 358 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [24304] 48 24304 85495 361 1 0 0 httpd Jul 3 02:18:45 web4 kernel: [24450] 27 24450 343008 697 0 0 0 mysqld Jul 3 02:18:45 web4 kernel: [24509] 48 24509 85495 365 3 0 0 httpd Jul 3 02:18:45 web4 kernel: [24894] 0 24894 19777 73 1 0 0 local Jul 3 02:18:45 web4 kernel: [24905] 0 24905 19777 118 0 0 0 local Jul 3 02:18:45 web4 kernel: [24950] 48 24950 85528 374 1 0 0 httpd Jul 3 02:18:45 web4 kernel: [24952] 48 24952 85495 357 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [24955] 48 24955 85495 357 4 0 0 httpd Jul 3 02:18:45 web4 kernel: [24973] 48 24973 85495 350 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25108] 48 25108 85495 389 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25139] 0 25139 82029 2864 0 0 0 rhn_check Jul 3 02:18:45 web4 kernel: [25159] 89 25159 19872 4 0 0 0 cleanup Jul 3 02:18:45 web4 kernel: [25226] 89 25226 19755 108 4 0 0 trivial-rewrite Jul 3 02:18:45 web4 kernel: [25243] 48 25243 85528 401 2 0 0 httpd Jul 3 02:18:45 web4 kernel: [25244] 48 25244 85495 355 1 0 0 httpd Jul 3 02:18:45 web4 kernel: [25271] 606 25271 2307 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [25294] 606 25294 56915 22255 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25308] 0 25308 19777 2 4 0 0 local Jul 3 02:18:45 web4 kernel: [25342] 48 25342 85495 363 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25369] 0 25369 35029 9 0 0 0 crond Jul 3 02:18:45 web4 kernel: [25392] 48 25392 85495 347 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25393] 0 25393 34474 4180 0 0 0 backup.pl Jul 3 02:18:45 web4 kernel: [25410] 636 25410 2205 15 1 0 0 procmail Jul 3 02:18:45 web4 kernel: [25435] 89 25435 19871 141 1 0 0 cleanup Jul 3 02:18:45 web4 kernel: [25441] 549 25441 2205 2 0 0 0 procmail Jul 3 02:18:45 web4 kernel: [25445] 557 25445 2205 47 1 0 0 procmail Jul 3 02:18:45 web4 kernel: [25453] 549 25453 2205 39 3 0 0 procmail Jul 3 02:18:45 web4 kernel: [25475] 549 25475 6392 2 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25476] 549 25476 2307 2 6 0 0 sh Jul 3 02:18:45 web4 kernel: [25477] 549 25477 57618 21448 4 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25480] 513 25480 2205 2 3 0 0 procmail Jul 3 02:18:45 web4 kernel: [25511] 513 25511 6392 12 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25536] 0 25536 4324 6 2 0 0 anacron Jul 3 02:18:45 web4 kernel: [25537] 513 25537 2307 6 7 0 0 sh Jul 3 02:18:45 web4 kernel: [25538] 0 25538 23948 105 0 0 0 sshd Jul 3 02:18:45 web4 kernel: [25540] 513 25540 57651 38995 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25573] 557 25573 2205 48 1 0 0 procmail Jul 3 02:18:45 web4 kernel: [25585] 536 25585 2205 26 1 0 0 procmail Jul 3 02:18:45 web4 kernel: [25590] 568 25590 2205 28 0 0 0 procmail Jul 3 02:18:45 web4 kernel: [25602] 0 25602 27116 144 3 0 0 bash Jul 3 02:18:45 web4 kernel: [25608] 536 25608 2205 28 0 0 0 procmail Jul 3 02:18:45 web4 kernel: [25633] 536 25633 6392 2 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25634] 536 25634 2307 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [25635] 536 25635 57652 48145 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25648] 536 25648 6392 6 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25660] 568 25660 6392 40 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25667] 536 25667 2307 2 7 0 0 sh Jul 3 02:18:45 web4 kernel: [25670] 536 25670 42888 33779 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25677] 568 25677 2307 2 0 0 0 sh Jul 3 02:18:45 web4 kernel: [25678] 568 25678 42761 34550 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25720] 48 25720 85495 381 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25732] 48 25732 85496 377 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25736] 48 25736 85495 373 4 0 0 httpd Jul 3 02:18:45 web4 kernel: [25753] 48 25753 85495 382 2 0 0 httpd Jul 3 02:18:45 web4 kernel: [25754] 48 25754 85495 376 2 0 0 httpd Jul 3 02:18:45 web4 kernel: [25755] 48 25755 85496 401 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25759] 48 25759 85495 398 5 0 0 httpd Jul 3 02:18:45 web4 kernel: [25762] 48 25762 85495 385 0 0 0 httpd Jul 3 02:18:45 web4 kernel: [25851] 513 25851 2205 32 0 0 0 procmail Jul 3 02:18:45 web4 kernel: [25886] 48 25886 85495 378 4 0 0 httpd Jul 3 02:18:45 web4 kernel: [25889] 48 25889 85495 368 1 0 0 httpd Jul 3 02:18:45 web4 kernel: [25891] 89 25891 24099 526 4 0 0 smtpd Jul 3 02:18:45 web4 kernel: [25901] 0 25901 35029 78 1 0 0 crond Jul 3 02:18:45 web4 kernel: [25902] 513 25902 6392 93 0 0 0 clam-wrapper.pl Jul 3 02:18:45 web4 kernel: [25903] 89 25903 19700 248 2 0 0 anvil Jul 3 02:18:45 web4 kernel: [25904] 513 25904 2307 37 0 0 0 sh Jul 3 02:18:45 web4 kernel: [25905] 513 25905 25794 20638 0 0 0 clamscan Jul 3 02:18:45 web4 kernel: [25906] 0 25906 32509 7803 4 0 0 monitor.pl Jul 3 02:18:45 web4 kernel: [25917] 89 25917 19873 420 1 0 0 cleanup Jul 3 02:18:45 web4 kernel: [25924] 48 25924 85495 377 3 0 0 httpd Jul 3 02:18:45 web4 kernel: [25959] 636 25959 2205 15 0 0 0 procmail Jul 3 02:18:45 web4 kernel: [25960] 636 25960 22364 5941 0 0 0 spamassassin Jul 3 02:18:45 web4 kernel: [25961] 48 25961 85495 393 1 0 0 httpd Jul 3 02:18:45 web4 kernel: [25962] 0 25962 19777 384 3 0 0 local Jul 3 02:18:45 web4 kernel: [26006] 48 26006 85495 389 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26007] 48 26007 85495 407 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26013] 89 26013 19873 447 0 0 0 cleanup Jul 3 02:18:48 web4 kernel: [26015] 89 26015 24099 549 6 0 0 smtpd Jul 3 02:18:48 web4 kernel: [26036] 557 26036 6392 171 0 0 0 clam-wrapper.pl Jul 3 02:18:48 web4 kernel: [26053] 557 26053 2307 119 7 0 0 sh Jul 3 02:18:48 web4 kernel: [26054] 557 26054 30180 24347 6 0 0 clamscan Jul 3 02:18:48 web4 kernel: [26067] 568 26067 2205 119 0 0 0 procmail Jul 3 02:18:48 web4 kernel: [26068] 89 26068 19753 328 4 0 0 smtp Jul 3 02:18:48 web4 kernel: [26079] 549 26079 2205 28 5 0 0 procmail Jul 3 02:18:48 web4 kernel: [26080] 549 26080 24476 8068 0 0 0 spamassassin Jul 3 02:18:48 web4 kernel: [26082] 0 26082 2205 116 3 0 0 procmail Jul 3 02:18:48 web4 kernel: [26087] 0 26087 2205 37 4 0 0 procmail Jul 3 02:18:48 web4 kernel: [26088] 0 26088 28918 19054 0 0 0 lookup-domain.p Jul 3 02:18:48 web4 kernel: [26092] 568 26092 6392 172 4 0 0 clam-wrapper.pl Jul 3 02:18:48 web4 kernel: [26096] 557 26096 6392 171 0 0 0 clam-wrapper.pl Jul 3 02:18:48 web4 kernel: [26101] 568 26101 2307 120 0 0 0 sh Jul 3 02:18:48 web4 kernel: [26102] 557 26102 2307 119 5 0 0 sh Jul 3 02:18:48 web4 kernel: [26106] 89 26106 19753 328 0 0 0 smtp Jul 3 02:18:48 web4 kernel: [26107] 568 26107 27649 22480 0 0 0 clamscan Jul 3 02:18:48 web4 kernel: [26108] 48 26108 85462 409 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26112] 89 26112 19711 316 4 0 0 bounce Jul 3 02:18:48 web4 kernel: [26114] 557 26114 13813 9057 0 0 0 clamscan Jul 3 02:18:48 web4 kernel: [26115] 0 26115 2307 120 4 0 0 sh Jul 3 02:18:48 web4 kernel: [26116] 0 26116 35029 168 2 0 0 crond Jul 3 02:18:48 web4 kernel: [26118] 48 26118 85462 410 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26120] 48 26120 85462 386 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26123] 48 26123 85462 386 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26125] 0 26125 3342 133 2 0 0 ps Jul 3 02:18:48 web4 kernel: [26129] 48 26129 85462 386 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26130] 48 26130 85462 392 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26131] 48 26131 85462 386 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26133] 48 26133 85462 387 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26134] 48 26134 85462 386 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26135] 48 26135 85462 391 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26136] 0 26136 14288 5460 0 0 0 monitor.pl Jul 3 02:18:48 web4 kernel: [26147] 0 26147 26399 5344 0 0 0 miniserv.pl Jul 3 02:18:48 web4 kernel: [26160] 606 26160 2205 119 0 0 0 procmail Jul 3 02:18:48 web4 kernel: [26167] 48 26167 85495 391 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26168] 48 26168 85462 382 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26169] 48 26169 85462 349 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26170] 48 26170 85495 390 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26171] 48 26171 85462 372 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26172] 48 26172 85462 395 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26173] 48 26173 85495 416 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26174] 48 26174 85462 388 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26180] 606 26180 6392 172 0 0 0 clam-wrapper.pl Jul 3 02:18:48 web4 kernel: [26183] 48 26183 85462 349 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26184] 606 26184 2307 120 1 0 0 sh Jul 3 02:18:48 web4 kernel: [26185] 606 26185 10550 5990 0 0 0 clamscan Jul 3 02:18:48 web4 kernel: [26188] 0 26188 28901 7882 0 0 0 miniserv.pl Jul 3 02:18:48 web4 kernel: [26189] 48 26189 85462 402 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26190] 48 26190 85462 382 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26191] 48 26191 85462 411 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26219] 48 26219 85462 388 3 0 0 httpd Jul 3 02:18:48 web4 kernel: [26220] 48 26220 85462 387 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26225] 89 26225 19711 312 0 0 0 bounce Jul 3 02:18:48 web4 kernel: [26226] 48 26226 85462 389 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26227] 48 26227 85462 400 1 0 0 httpd Jul 3 02:18:48 web4 kernel: [26229] 48 26229 85462 383 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26230] 48 26230 85462 380 2 0 0 httpd Jul 3 02:18:48 web4 kernel: [26231] 48 26231 85462 384 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26232] 48 26232 85462 372 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26233] 48 26233 85462 378 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26239] 89 26239 24066 531 0 0 0 smtpd Jul 3 02:18:48 web4 kernel: [26241] 0 26241 19777 372 5 0 0 local Jul 3 02:18:48 web4 kernel: [26242] 0 26242 2205 115 1 0 0 procmail Jul 3 02:18:48 web4 kernel: [26243] 606 26243 2205 120 1 0 0 procmail Jul 3 02:18:48 web4 kernel: [26249] 0 26249 2205 115 0 0 0 procmail Jul 3 02:18:48 web4 kernel: [26258] 0 26258 19147 185 0 0 0 sendmail Jul 3 02:18:48 web4 kernel: [26260] 48 26260 85462 380 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26261] 568 26261 4791 37 2 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26262] 48 26262 85462 348 4 0 0 httpd Jul 3 02:18:48 web4 kernel: [26263] 48 26263 85462 383 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26265] 626 26265 16000 96 1 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26266] 626 26266 4791 38 5 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26267] 626 26267 5435 38 0 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26268] 568 26268 5435 38 1 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26269] 626 26269 16000 97 7 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26270] 626 26270 16000 96 5 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26271] 568 26271 4791 37 2 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26272] 626 26272 4791 37 6 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26273] 568 26273 5435 39 5 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26274] 606 26274 4791 38 2 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26275] 626 26275 4791 37 7 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26276] 568 26276 16000 98 2 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26277] 626 26277 4791 38 3 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26278] 568 26278 16000 96 3 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26279] 626 26279 4791 37 2 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26280] 606 26280 4791 38 3 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26281] 568 26281 4791 38 6 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26282] 553 26282 4791 38 6 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26283] 568 26283 4791 38 7 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26284] 568 26284 4791 38 5 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26285] 568 26285 16000 99 0 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26286] 568 26286 5435 38 4 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26287] 626 26287 16000 96 5 0 0 php-cgi Jul 3 02:18:48 web4 kernel: [26290] 0 26290 38845 2085 0 0 0 lfd Jul 3 02:18:48 web4 kernel: [26292] 0 26292 2307 126 0 0 0 sh Jul 3 02:18:48 web4 kernel: [26293] 0 26293 19682 71 0 0 0 master Jul 3 02:18:48 web4 kernel: [26297] 0 26297 85463 328 0 0 0 httpd Jul 3 02:18:48 web4 kernel: [26301] 0 26301 2091 86 0 0 0 diff Jul 3 02:18:48 web4 kernel: [26302] 606 26302 4807 55 0 0 0 clam-wrapper.pl Jul 3 02:18:48 web4 kernel: Out of memory: Kill process 23015 (clamscan) score 69 or sacrifice child Jul 3 02:18:48 web4 kernel: Killed process 23015, UID 557, (clamscan) total-vm:230476kB, anon-rss:52024kB, file-rss:288kB
While copying and pasting the output I found that 2 out of 3 backups are still running. Which is really weird. Normally, they would be done by now according to previous backups logs. But how do backups cause clamscan to totally freak out?
Thanks again,
Dave.