Server Real Memory and Virtual Memory peaks and MySQL server goes down

Hello,

my server has continuous peaks in real memory (I have 4GB ram) and virtual memory (1.94 GB), and I mean almost every time I load a page! The websites were faster when I was on a shared host, I’m really going crazy!

I installed atop to see what was causing the server load, and this is the result of the command atop -r (I removed the end tail with all the processes that wheren’t taking any cpu nor memory load):

ATOP - mail 2014/04/14 10:51:58 --------- 10m0s elapsed

PRC | sys 2m05s | user 7m53s | #proc 172 | #zombie 0 | #exit 231 |

CPU | sys 20% | user 65% | irq 0% | idle 14% | wait 1% |

CPL | avg1 46.10 | avg5 23.70 | avg15 14.49 | csw 555283 | intr 538460 |

MEM | tot 3.7G | free 265.4M | cache 200.8M | buff 1.4M | slab 64.2M |

SWP | tot 1.9G | free 1.6G | | vmcom 6.4G | vmlim 3.8G |

PAG | scan 290944 | steal 187743 | stall 3406 | swin 5936 | swout 73014 |

LVM | roup-lv_root | busy 5% | read 14088 | write 8225 | avio 1.46 ms |

LVM | roup-lv_swap | busy 2% | read 5936 | write 73014 | avio 0.13 ms |

DSK | sda | busy 7% | read 13624 | write 3368 | avio 2.32 ms |

NET | transport | tcpi 6934 | tcpo 5793 | udpi 33 | udpo 33 |

NET | network | ipi 7353 | ipo 5920 | ipfrw 0 | deliv 6967 |

NET | eth0 0% | pcki 7305 | pcko 7716 | si 12 Kbps | so 112 Kbps |

NET | lo ---- | pcki 615 | pcko 615 | si 0 Kbps | so 0 Kbps |

PID SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPU CMD
15633 1m50s 2m55s 2.4G 658.4M 142.2M 142.7M – - S 47% mysqld
15040 1.12s 35.35s 0K 0K - - NE 0 E 6%
15615 0.84s 29.38s 0K 0K - - NE 0 E 5%
15616 0.90s 29.21s 0K 0K - - NE 0 E 5%
15591 0.18s 6.47s 0K 0K - - -E 0 E 1%
16835 0.39s 5.91s 0K 0K - - NE 0 E 1%
15356 0.15s 5.26s 0K 0K - - -E 0 E 1%
15280 0.12s 4.92s 0K 0K - - -E 0 E 1%
16911 0.32s 4.57s 1.0G 203.6M 668K 4K N- - S 1% httpd
16910 0.21s 4.43s 944.8M 173.1M 388K 0K N- - S 1% httpd
15262 0.09s 4.55s 0K 0K - - -E 0 E 1%
16833 0.26s 4.35s 0K 0K - - NE 0 E 1%
16837 0.17s 4.00s 0K 0K - - NE 0 E 1%
15271 0.12s 4.04s 0K 0K - - -E 0 E 1%
15285 0.07s 3.99s 0K 0K - - -E 0 E 1%
16907 0.26s 3.78s 852.1M 58520K 1212K 8K N- - S 1% httpd
15273 0.10s 3.91s 0K 0K - - -E 0 E 1%
15272 0.10s 3.76s 0K 0K - - -E 0 E 1%
16832 0.36s 3.49s 0K 0K - - NE 0 E 1%
15380 0.08s 3.66s 0K 0K - - -E 0 E 1%
16834 0.24s 3.48s 0K 0K - - NE 0 E 1%
14618 0.10s 3.49s 0K 0K - - -E 0 E 1%
16914 0.09s 3.48s 825.0M 60692K 120K 192K N- - S 1% httpd
15277 0.14s 3.27s 0K 0K - - -E 0 E 1%
16922 0.09s 3.30s 824.4M 62512K 68K 12K N- - S 1% httpd
15593 0.09s 3.27s 0K 0K - - -E 0 E 1%
15432 0.09s 3.07s 0K 0K - - -E 0 E 1%
16855 0.09s 3.01s 0K 0K - - NE 0 E 1%
16861 0.07s 2.81s 0K 0K - - NE 0 E 0%
16927 0.09s 2.71s 826.3M 65132K 456K 124K N- - R 0% httpd
16860 0.08s 2.63s 0K 0K - - NE 0 E 0%
15507 0.08s 2.55s 0K 0K - - -E 0 E 0%
15424 0.06s 2.52s 0K 0K - - -E 0 E 0%
16940 0.08s 2.47s 826.7M 65492K 0K 4K N- - S 0% httpd
16912 0.11s 2.43s 832.4M 67508K 160K 0K N- - S 0% httpd
16937 0.10s 2.43s 827.3M 63572K 364K 4K N- - S 0% httpd
15260 0.07s 2.41s 0K 0K - - -E 0 E 0%
16858 0.13s 2.31s 0K 0K - - NE 0 E 0%
16839 0.10s 2.33s 0K 0K - - NE 0 E 0%
16854 0.09s 2.21s 0K 0K - - NE 0 E 0%
16928 0.10s 2.15s 833.2M 75508K 132K 0K N- - S 0% httpd
16849 0.10s 1.90s 0K 0K - - NE 0 E 0%
16926 0.09s 1.89s 831.4M 72836K 128K 0K N- - S 0% httpd
16840 0.07s 1.87s 0K 0K - - NE 0 E 0%
16842 0.11s 1.80s 0K 0K - - NE 0 E 0%
16936 0.09s 1.80s 831.2M 72528K 232K 4K N- - S 0% httpd
16919 0.11s 1.72s 825.0M 64140K 168K 72K N- - S 0% httpd
16934 0.09s 1.72s 830.9M 71904K 320K 0K N- - S 0% httpd
16836 0.08s 1.70s 0K 0K - - NE 0 E 0%
16925 0.05s 1.66s 827.2M 66752K 32K 12K N- - S 0% httpd
16913 0.07s 1.60s 829.2M 66484K 308K 4K N- - S 0% httpd
16939 0.08s 1.56s 830.4M 70904K 288K 0K N- - S 0% httpd
16859 0.07s 1.53s 0K 0K - - NE 0 E 0%
16921 0.09s 1.50s 826.4M 67324K 20K 84K N- - S 0% httpd
16924 0.08s 1.50s 826.4M 66224K 16K 12K N- - S 0% httpd
16841 0.08s 1.50s 0K 0K - - NE 0 E 0%
15287 0.05s 1.50s 0K 0K - - -E 0 E 0%
16844 0.10s 1.32s 0K 0K - - NE 0 E 0% <miniserv.pl>
16920 0.06s 1.27s 826.2M 64336K 60K 80K N- - S 0% httpd
16929 0.08s 1.20s 831.7M 71104K 0K 0K N- - S 0% httpd
16909 0.07s 1.21s 826.4M 52852K 68K 92K N- - S 0% httpd
16974 0.10s 1.17s 825.0M 75576K 68K 80K N- - S 0% httpd
16999 0.08s 1.18s 824.4M 71132K 136K 4K N- - S 0% httpd
16938 0.07s 1.12s 825.9M 64984K 96K 0K N- - S 0% httpd
17090 0.07s 1.10s 826.2M 64128K 364K 4K N- - S 0% httpd
16970 0.06s 1.08s 826.0M 64008K 224K 0K N- - S 0% httpd
16985 0.04s 1.10s 825.2M 63028K 0K 0K N- - S 0% httpd
17080 0.05s 1.08s 826.2M 64420K 64K 4K N- - S 0% httpd
17001 0.07s 1.05s 827.2M 64008K 7276K 8K N- - S 0% httpd
16843 0.07s 1.05s 0K 0K - - NE 0 E 0%
17007 0.04s 1.05s 826.7M 64772K 436K 4K N- - S 0% httpd
17019 0.06s 0.99s 826.7M 64736K 100K 4K N- - S 0% httpd
16845 0.08s 0.95s 0K 0K - - NE 0 E 0% <miniserv.pl>
17004 0.11s 0.90s 835.7M 84168K 108K 8K N- - S 0% httpd

As far as I can see there’s one big process with mysql happening at the beginning:

15633 1m50s 2m55s 2.4G 658.4M 142.2M 142.7M – - S 47% mysqld

When I went to check MySQL Database Server, it was down and then restarted by itself. So I went to see all executing processes, filtered then by memory usage and the first one was that big mysqld process occupying more than 3 GB of memory! When I opened it up, this was the command:

/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

I only have one big MySQL database of 340 MB and I really don’t understand how it could occupy 3GB of ram everytime a page is loaded.

Could you please help me nailing down to the problem? Search engines are not indexing my pages becuase they’re all timing out, it’s 4 days fulltime that I’m trying to resolve this, it’s becoming a nightmare!

Please help, Thanks again

This appears to be a duplicate of this post over here:

https://virtualmin.com/node/32764