Services Are Down (BIND DNS, Postfix Mail, MySQL). Need Urgent Help please...

Ahead of time, i am a linux noob. So if you want me to do something, please tell me how to do it also please :slight_smile:

I am in need of urgent help. Unsure whats going on with my server. Last night all my websites and web services went down on my vps, i contacted my host (volumedrive) and they changed the vps over to new hardware etc. So now the vps is on new hardware and im still getting the same issue/error (basically it didnt fix anything).

Look at the attached image (1.jpg - https://www.virtualmin.com/system/files/1_0.JPG), it shows my virtualmin system information page. it shows that 3 of the services are down and i cant seem to turn them back on (they were running fine a day ago). Can anyone help me? I need to know whats causing this (weather it is Virtualmin, Any other server/linux issue, or my hosts fault)ā€¦

This is what i currently get:

When i click ā€œrefresh/recheck configurationsā€ i get this:

The status of your system is being checked to ensure that all enabled features are available, that the mail server is properly configured, and that quotas are active .. BIND DNS server is installed, and the system is configured to use it. The Postfix server does not appear to be installed on your system, or has not yet been set up properly in Webmin's Postfix Configuration module. If you are running a different mail server, select it on the module configuration page.

ā€¦ your system is not ready for use by Virtualmin.

if i try to start BIND DNS Server i get:
Failed to start service : Failed to start BIND : Starting named: [FAILED]

if i try to start Postfix Mail Server i get:
Failed to start service : postsuper: fatal: scan_dir_push: open directory defer: Permission denied

if i try to start MySQL Database Server i get:
Failed to start service :
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL: [FAILED]

You could try updating them packages you have 40 lol there
Also run the re-check

did that earlier alreadyā€¦ didnt change anythingā€¦ :frowning:

Iā€™d also suggest to first take a look at the package manager (whichever comes with CentOS). For one, to perform the updates, and then, to check for broken dependencies or broken packages. Also try manually starting the BIND and MySQL services in SSH.

Vmin complaining about a missing Postfix and unable to start BIND and MySQL sounds like something is rather seriously broken thereā€¦ What exactly can have happened that caused this? Some system crash or forced reboot or something?

Do you have backups available (of the system or websites)? Itā€™s possible that you need to reinstall the system and restore a backup, especially if youā€™re a ā€œLinux noobā€ and donā€™t really know how to fix stuff. :slight_smile:

The volumedrive customer support is very poor. They just sent me the following email after about 12 hours of bugging them:

After looking over the log files and trying to get the services to start, we believe the best suggestion would be to create a new account and have everything moved to the new VPS. We apologize that some of the process and files were damaged in the move, but it was necessary. The IP will stay the same for the VPS. If you would like to do this, please advise and we will have the new VPS setup ASAP so you can begin moving the files.

Now i have no prior backups of the linux system or of virtualmin. Any suggestions on how i can transfer the data over?

For one, i cannot make a backup of the virtualmin as it for obvious reasons:

Backup failed : DBI connect failed : Canā€™t connect to local MySQL server through socket ā€˜/var/lib/mysql/mysql.sockā€™ (2)

what can i do?

Locutus, thanks for the info. but i have no idea how to do the things you told meā€¦ :frowning:
And i have no idea what originally caused this, they just completely ignore some of my questions, and im getting a bit annoyed at them for itā€¦ :S

Im kinda screwed at the moment as i had few client websites running on the server (and they were using the MySQL databases)ā€¦

Thanks. Yeah, im very new to this as you could probably guess. I have googled and figured out how to use bind.
I tried running Bind:

/etc/init.d/named start
Starting named: [FAILED]

Does this mean anything?

Question: what would happen if i tried to reinstall bind again using ssh? Is that a bad idea?

Well, thatā€™s surely a pityā€¦ But I must say that creating regular backups is one of the first and foremost "must"s for a server administrator (especially since Virtualmin and Webmin have excellent backup features). Things can basically go awfully wrong at any time, as youā€™ve seen nowā€¦

(Aside from this, Iā€™d quite probably ditch that ā€œhosterā€ youā€™re usingā€¦ If theyā€™re unable to move your virtual machine to another host without breaking most of your installed services, however they managed to do that, and donā€™t even hold backups of those for youā€¦ Not really the service Iā€™d be using. :slight_smile: )

To save the databases now, it might help to copy away the /var/lib/mysql directory, where MySQL should - at least it does so in Ubuntu - store all its databases. You might be able to copy those files back after the reinstall.

Same goes for the contents of /home, where all vserver files, users and mails should be stored.

And you might copy away /etc, which holds all config files. Warning! Do not simply copy back the contents of /etc after the reinstall, but only use the copy as reference, in case the new installation is missing things or settings.

As for ā€œno idea how to do the things I told youā€: Iā€™m sorry, but frankly I must say, if you donā€™t know how to login to your server via SSH and restart a service (and also cannot use Google etc. to find out how to do that), Iā€™m afraid Iā€™ll have to suggest you should reconsider if operating a Linux-based hosting server is the right thing for youā€¦ Virtualmin is a great help for web hosting, and eases mostly all processes involved in that, but it generally does not liberate you from the responsibility to still have at least a basic understanding of the underlying system.

Hehe, not very talkative there. Youā€™ll probably find more in /var/log/syslog as to why the startup failed.

Reinstalling the package sure canā€™t hurt, as in it probably wonā€™t damage more than is already broken. :wink: Unfortunately though I canā€™t help you with operating the CentOS package manager, since Iā€™m using Ubuntu (a Debian derivative) myself, which has a quite different one.

Sigh, reinstalling bind didnt make any difference.

I asked the host to set me up with a new VPS, now i need to reinstall everything and transfer things over.

Is there a easy way to transfer files from 1 VPS to another using ssh if i know the ip addresses of both?

ps. yeah im sooo tempted to move hosts nowā€¦ siiighā€¦

Yeah, you can use SFTPā€¦ Which is like an FTP client, but using the SSH protocol to transfer stuff. Thereā€™s a text mode client ā€œSFTPā€ which works much like the Linux FTP, albeit itā€™s not very comfortable. Might use the Midnight Commander instead which can make sn SFTP connection using ā€œShell Linkā€ in the Left/Right menu. Might have to install the package for that though, which is named mc in Ubuntu.

OK, now this is where im uptoā€¦

New VPS has everything installed virtualmin working etcā€¦

But now, i need to migrate the database over from the old one.
as Locutus mentioned, i copied over all the files in:
/var/lib/mysql
to the same directory on the new server.

But that seems to have screwed up a lot of things maybe? This database has a joomla website by the way.
Virtualmin says the following:

Database name db_test
Database server type MySQL
Size on disk 1.14 MB
Number of tables 46

seems like there are some content. But if i click manage this database i get:
SQL show index from jos_banner failed : Canā€™t find file: ā€˜./db_test/jos_banner.frmā€™ (errno: 13)
This file is actually in the server.

can anyone understand why this is happening?

Oh also, i did make the same database names on virtualmin before i copied files over.

and now if i try to create a new database (db name = bob) i get:

Failed to create database : SQL insert into db (host, db, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Execute_priv) values ('localhost', 'bob', 'gismlk', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y') failed : Duplicate entry 'gismlk' for key 2

Did you also copy all subdirectories from /var/lib/mysql? Did you stop MySQL while copying the files, and restart it afterwards?

yes, yes and yesā€¦ :frowning:

now if this method wont work, is there any other way to export a mysql database without mysql running? O.o

siigh, this is hurting my brain lolā€¦

now if this method wont work, is there any other way to export a mysql database without mysql running?

Without MySQL running, youā€™d need to just copy the database files. Thatā€™s a fast and reliable way of copying MySQL databases around.

Do you see the directory ā€œ/var/lib/mysql/db_testā€ on both servers?

And then in that directory should be the file ā€œjos_banner.frmā€, does that exist on both servers?

Next, youā€™d want to verify that the permissions and ownership are the same in both cases. You can see the permissions and owners by running ā€œls -l /var/lib/mysql/db_test/jos_banner.frmā€.

-Eric

Ownership and permission is a good point, indeed. Although then the error message should have been ā€œpermission deniedā€, and not ā€œcanā€™t find fileā€. :slight_smile:

Next step, if the jos_banner.frm file does exist in the old and new database directory, would be to test accessing it directly from the MySQL command line client.

(If the file actually does not exist, it probably got corrupted/deleted during whatever damaged the rest of your server. At which point youā€™d again need that backupā€¦ :frowning: )

Execute this in SSH:

mysql -p{your-mysql-root-password}

Make sure thereā€™s no space between the -p and the password, and itā€™s not your server root password, but the one you entered during the ā€œVirtualmin post-install wizardā€ as your MySQL root password. Then execute:

show databases; use db_test; show tables; select * from jos_banner limit 5;

If it works okay, that will show you all databases on the server, then all tables in the database ā€œdb_testā€, then the first 5 rows in the ā€œjos_bannerā€ table.

ok, i see the problem hereā€¦ when i execute those mysql commands it says:

mysql> show tables;
ERROR 1018 (HY000): Canā€™t read dir of ā€˜./db_test/ā€™ (errno: 13)

then i checked the permissionsā€¦ the permissions of the contents of the both folders (new and old) are both exactly the same. BUT, the permissions of the folder it self is differentā€¦

OLD
drwx------ 2 root root 4096 Nov 10 14:03 db_test

NEW
drwsā€“Sā€” 2 root my_user2 4096 Nov 11 15:01 db_test

now to figure out how to fix permissionsā€¦ argā€¦
do you think this is the cause? My understanding of these permission letters are minimal, but i wonder what that S representā€¦ :S

Yeah, both of those owner/permission sets on the old and new server look odd. Normally, the database subdirectories (except for ā€œmysqlā€ itself) are owned by mysql:{virtualserveradmingroup} and have permissions like those on your new server. If that differs, it can clearly be the reason. In your case, the MySQL process does not get access rights cause the directory is owned by root.

You should check under which user account the MySQL daemon is running, you can do that with ps aux | grep mysql and check for the line with the mysqld executable. The user (first column) listed there should be the owner of database directories/files.

Ownership can be changed with chown, permissions with chmod.

The ā€œsā€ bit is called ā€œsetuid / setgidā€ (set user ID, set group ID). They cause that new files created inside the directory get the ownership of the directory, and not (as is normally the case) the ownership of the user creating the file.

are you by anychance able to give me a ls -l of your mysql folder, so i know what its supposed to look like?

ill try doing what you said for the time being :slight_smile:

YEY! its finally fixedā€¦ oh god that took a whileā€¦

basically had to reset the owners on the databases and the tablesā€¦ Then it was throwing an error saying some tables dont existā€¦ then i had to go manually delete these tables and any links they had from memory 1 by 1ā€¦ hmmmm, and that fixed itā€¦ :slight_smile:

Thank you both for your awesome helpā€¦ ahhhh, now i can go get some sleep XDā€¦

btw, anyone know a good/cheap backup service i can use easily with virtualmin? O.o (not going through this againā€¦ everā€¦)ā€¦ lolā€¦

btw, anyone know a good/cheap backup service i can use easily with virtualmin?

If you havenā€™t already, you may want to take a look at Virtualminā€™s built-in backups. You can access them by going into Backup and Restore -> Scheduled Backups.

Iā€™d suggest setting up a backup to run each day.

You can have it store the backup locally, or on a remote FTP or SSH server.

-Eric