Help with troubleshooting problems please

Brook · June 10, 2012, 8:58pm

My server’s acting a bit weird. It’s a Virtualmin install on a minimal CentOS 6.2 system (the only thing I needed to set before the VM install was the hostname). Installed about 10 days ago.

Last night md2_raid1 was taking up about 15% of cpu and the loads were higher than normal for about 40 minutes or so. I checked both drives (Raid 1) via the smart monitor in Webmin and no errors were reported. In the end I just rebooted the server and it seemed fine after that.

Then about an hour ago I went on one of my sites and it was not resolving, looking at top stats it wasn’t showing any php-cgi connections to any of my sites. So I restarted bind, but still nothing. So I rebooted the server and it’s all working again now.

Any ideas how to troubleshoot what’s going on? Or how I can check that raid is set-up correctly?

Here’s the output from var/log/messages - the problem seems to have occurred at around 20:13

Pastie 1: http://pastie.org/4063664
Pastie 2: http://pastie.org/4063677

Thanks in advance for any help.

Eric · June 11, 2012, 2:19pm

Howdy,

It’s possible that the RAID somehow got out of sync, and that it was simply being corrected.

What does this command show:

cat /proc/mdstat

Also, what about this command:

dmesg | tail -30

Brook · June 11, 2012, 3:01pm

cat /proc/mdstat


Personalities : [raid1] 
md1 : active raid1 sdb2[1] sda2[0]
      524276 blocks super 1.0 [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]

2096116 blocks super 1.0 [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]

729952120 blocks super 1.0 [2/2] [UU]

(Although when I did that the night it had problems md0 had lots of numbers after the UU (not sure now if it was UU or U_))

dmesg | tail -30


Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=29234 DF PROTO=TCP SPT=44771 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=6110 DF PROTO=TCP SPT=44771 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=24535 DF PROTO=TCP SPT=19787 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=10751 DF PROTO=TCP SPT=19787 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=21940 DF PROTO=TCP SPT=19787 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=23219 DF PROTO=TCP SPT=9610 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=11414 DF PROTO=TCP SPT=9610 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=19949 DF PROTO=TCP SPT=9610 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=16202 DF PROTO=TCP SPT=4682 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=6737 DF PROTO=TCP SPT=4682 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=18039 DF PROTO=TCP SPT=4682 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=14713 DF PROTO=TCP SPT=42727 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=25067 DF PROTO=TCP SPT=42727 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=10099 DF PROTO=TCP SPT=34998 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=31297 DF PROTO=TCP SPT=34998 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=210.213.141.136 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=12219 DF PROTO=TCP SPT=34998 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=190.68.238.100 DST=my.ser.ver.ip LEN=433 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=5061 DPT=5060 LEN=413 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=19416 DF PROTO=TCP SPT=58183 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=19201 DF PROTO=TCP SPT=58183 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=21880 DF PROTO=TCP SPT=58183 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=9954 DF PROTO=TCP SPT=48812 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=10470 DF PROTO=TCP SPT=48812 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=12762 DF PROTO=TCP SPT=48812 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=16554 DF PROTO=TCP SPT=6092 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=17345 DF PROTO=TCP SPT=6092 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=21341 DF PROTO=TCP SPT=6092 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=15069 DF PROTO=TCP SPT=14880 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=21994 DF PROTO=TCP SPT=32952 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=15097 DF PROTO=TCP SPT=38898 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0 
Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=6c:62:6d:85:b8:20:2c:6b:f5:8a:43:00:08:00 SRC=116.50.142.29 DST=my.ser.ver.ip LEN=48 TOS=0x00 PREC=0x00 TTL=104 ID=15714 DF PROTO=TCP SPT=38898 DPT=46 WINDOW=65535 RES=0x00 SYN URGP=0

Since then I have done a yum update, and noticed the kernal was updated:

Package Arch Version Repository Size

Installing:
kernel x86_64 2.6.32-220.17.1.el6 updates 24 M
Updating:
kernel-firmware noarch 2.6.32-220.17.1.el6 updates 6.3 M
tzdata noarch 2012c-1.el6 updates 441 k
virtualmin-release noarch 1.0-5.gpl.rhel virtualmin 10 k

Maybe that’ll fix it? (It’s been fine since, touch wood!)

Eric · June 11, 2012, 4:03pm

Yeah, what you’re describing makes it sound like the RAID somehow got out of sync.

I’ve seen that happen before in cases where nothing was wrong with the disk. So what is the problem then? I’m not quite sure!

But hopefully, that won’t be a frequent occurrence Especially with that kernel update.

-Eric

Brook · June 11, 2012, 5:06pm

Thanks Eric - I hope it’s the last of it too

Brook · June 17, 2012, 2:14pm

It’s happening again

Is it common for it to resync so often? Could there be a problem I need to look into?

Here’s the results from cat /proc/mdstat:

Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[0]
524276 blocks super 1.0 [2/2] [UU]

md0 : active raid1 sdb1[1] sda1[0]
2096116 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid1 sdb3[1] sda3[0]
729952120 blocks super 1.0 [2/2] [UU]
[======>…] check = 32.6% (238493184/729952120) finish=133.7min speed=61232K/sec

Is it possible the datacenter set it up incorrectly? Any ideas how to check that it’s all set-up ok?

Thanks in advance.

Edit: Just to add I left it overnight and it went through the resync and is fine today. Are resyncs regular occurrences? I don’t remember it happening on my last server.

helpmin · June 17, 2012, 3:16pm

yes, checks usually happen once a week on centos 6. you can probably disable the cron job, but this doesn’t mean I recommended this

Brook · June 17, 2012, 11:58pm

Thanks, if this is normal then I shall leave it be