Scheduled backups aren't triggered

ipsa · June 2, 2009, 2:06pm

Hi,

I’ve setup scheduled backups (full monthly, incremental daily) on my server and they don’t seem to get triggered.

The root user’s crontab contains this:

@monthly /etc/webmin/virtual-server/backup.pl --id 123974394718634
@daily /etc/webmin/virtual-server/backup.pl --id 123974406619089

and if I execute either one of those commands manually at the command line, the backups gets done as expected. Same if I do it manually in the Virtualmin web interface.

I’m running Virtualmin 3.68 Pro on CentOS Linux 5.3.

Is there a problem with crontab? Maybe the "@word" notation?

Thanks

Eric · June 2, 2009, 2:23pm

Howdy,

Well, those 2 entries look okay.

What happens if you restart cron, then look in /var/log/messages – do you see any errors in there?

It’s possible that an error with a different cron entry is causing cron in general to bail out.
-Eric

ipsa · June 2, 2009, 4:48pm

I did “service crond restart” and nothing got added to /var/log/messages. I’m assuming that means that the restart occurred without any error.

Eric · June 2, 2009, 5:19pm

Well, you should see at least a message stating that it restarted.

If you don’t see that, it’s in one of the other files there. But I forget which, I don’t have a CentOS machine right here
-Eric

Joe · June 2, 2009, 7:55pm

crond is the right service. I don’t see any messages when restarting it on my system, either. So, I guess we can’t use that as an indicator of anything.

ipsa · June 3, 2009, 11:43am

So, I’ve restarted the service again and this line got added to the /var/log/cron file:

Jun 3 11:36:30 server crond[9148]: (CRON) STARTUP (V5.0)

In the same file, there’s a whole bunch of messages like this:


Jun  3 00:00:01 server crond[3114]: Cannot make/remove an entry for the specified session
Jun  3 00:00:01 server crond[3114]: CRON (root) ERROR: failed to open PAM security session: Success
Jun  3 00:00:01 server crond[3114]: CRON (root) ERROR: cannot set security context
Jun  3 00:00:01 server crond[3116]: Cannot make/remove an entry for the specified session
Jun  3 00:00:01 server crond[3116]: CRON (mailman) ERROR: failed to open PAM security session: Success
Jun  3 00:00:01 server crond[3116]: CRON (mailman) ERROR: cannot set security context
Jun  3 00:00:01 server crond[3118]: Cannot make/remove an entry for the specified session
Jun  3 00:00:01 server crond[3118]: CRON (root) ERROR: failed to open PAM security session: Success
Jun  3 00:00:01 server crond[3118]: CRON (root) ERROR: cannot set security context
Jun  3 00:00:01 server crond[3122]: Cannot make/remove an entry for the specified session
Jun  3 00:00:01 server crond[3122]: CRON (root) ERROR: failed to open PAM security session: Success
Jun  3 00:00:01 server crond[3122]: CRON (root) ERROR: cannot set security context
Jun  3 00:00:01 server crond[3119]: Cannot make/remove an entry for the specified session
Jun  3 00:00:01 server crond[3119]: CRON (root) ERROR: failed to open PAM security session: Success
Jun  3 00:00:01 server crond[3119]: CRON (root) ERROR: cannot set security context

These are all from midnight last night, when it should have tried to to the incremental backup (and a whole bunch of other things, I assume). I’m assuming there’s a link but since CentOS isn’t my usual platform, I’ll be asking google after posting here.

Eric · June 3, 2009, 11:59am

Aha! I think you’re having an SELinux problem.

Now, the above should usually work – SELinux shouldn’t prevent cron jobs from running.

But, to simplify your life a bit – you might consider simply disabling SELinux.

Take a look in /etc/selinux/config – I would suggest setting "SELINUX=disabled" in there.

Also, to prevent having to reboot, you can run this (as root):

[code:1]echo 0 >/selinux/enforce[/code:1]

Which will manually switch it off – and the next time you reboot, the changes to the config file will prevent it from coming on at all.
-Eric

ipsa · June 3, 2009, 2:29pm

I’m all for simplifying my life but isn’t SELinux a good thing? I don’t know much about it but disabling it wholesale seems a little extreme. Does it not bring advantages that would outweigh a few minutes/hours spent configuring it correctly?

Anyway, I’ll do it if the other solution is really heinous and cludgy and SELinux isn’t worth it, but I’d like to know more before doing so.

Eric · June 3, 2009, 2:42pm

How about this – can you try disabling SELinux temporarily – and if that does indeed solve the problem, you can then decide what direction to move from there.

If it doesn’t solve the problem, well, then something else is hindering cron and the backups

To do that, just run:

echo 0 > /selinux/enforce
/etc/init.d/crond restart

Check out the cron log file, and see if those errors persist.

You can easily re-enable selinux with:

echo 1 > /selinux/enforce

Joe · June 3, 2009, 3:35pm

SELinux is a good thing, but the reliability of it and its policies on most distros currently leaves much to be desired. And, the tools for managing policy truly and completely suck.

Most importantly, though, not having cron is a pretty big problem, and you need to take steps to get the problem corrected.

If you know what you’re doing with SELinux (and are OK with users being pretty serious restricted, and occasional mysterious errors) then you could turn it back on after testing with it turned off. Virtualmin has the beginnings of support for setting context for files and such (though it’s still a long way from complete, so your policy will need to be pretty open for users).

ipsa · June 3, 2009, 5:33pm

Okay,

I tried to disable SELinux as suggested by running this:

echo 0 > /selinux/enforce

and then restarted crond. Those same message continue to appear in the cron log file.

It seems that by running the command above, I created a folder named “selinux” at the root and a file named “enforce” inside it with “0” as its content. (There’s nothing else in the folder and its date show as being a few moments ago.)

I understand from the notation that that’s what it is supposed to do, but I figured “selinux” was a special folder (like proc).

Is something else wrong here?

Joe · June 3, 2009, 6:00pm

Hehehe…Actually you can use:

setenforce 0

Not sure what Eric was after with the echo command…I guess there probably is a /proc entry for selinux. I dunno what it is, though.

Eric · June 3, 2009, 7:46pm

Hrm, when SELinux is enabled, there should be a /selinux directory that is a bit like /proc.

That not being there suggests to me that it’s either not enabled, or that my information is really old. Either is totally plausible
-Eric

Joe · June 3, 2009, 7:48pm

Actually, you’re right. There is an /selinux directory. There is no “enforce” in my system, though.

ipsa · June 4, 2009, 10:31am

Okay, SELinux is disabled.

[code:1]bash-3.2# setenforce 0
setenforce: SELinux is disabled[/code:1]

I then restarted cron and monitored the /var/log/cron file. Here’s its contents after a few minutes:

[code:1]Jun 4 10:22:26 server crond[26301]: (CRON) STARTUP (V5.0)
Jun 4 10:25:02 server crond[26880]: Cannot make/remove an entry for the specified session
Jun 4 10:25:02 server crond[26880]: CRON (mailman) ERROR: failed to open PAM security session: Success
Jun 4 10:25:02 server crond[26880]: CRON (mailman) ERROR: cannot set security context
Jun 4 10:25:02 server crond[26881]: Cannot make/remove an entry for the specified session
Jun 4 10:25:02 server crond[26881]: CRON (root) ERROR: failed to open PAM security session: Success
Jun 4 10:25:02 server crond[26881]: CRON (root) ERROR: cannot set security context
Jun 4 10:26:01 server crond[27254]: Cannot make/remove an entry for the specified session
Jun 4 10:26:01 server crond[27254]: CRON (root) ERROR: failed to open PAM security session: Success
Jun 4 10:26:01 server crond[27254]: CRON (root) ERROR: cannot set security context[/code:1]

This isn’t for the backups specifically, but something seems to still be wrong with cron.

Eric · June 4, 2009, 3:53pm

What distro/version are you using?

And is everything fully up to date – all the latest patches and updates made available by the vendor have been applied?

There’s some issues similar to what you’re seeing on some older versions of RHEL/CentOS/Fedora, but they all should have been resolved by the latest updates.
-Eric

ipsa · June 5, 2009, 11:06am

I’m running Virtualmin 3.68 Pro on CentOS Linux 5.3.

I’m not fully up to date. yum has a problem updating some clamav stuff (it seems to be looking for /usr/bin/freshclam, which is there but it’s still complaining that it is missing; wouldn’t mind pointer with that one either).

As a crutch until this cron issue is fixed, I’m using the cron on my local box to SSH into the server and run the scripts. It’s not pretty, but it works.

Eric · June 5, 2009, 11:50am

Well, as a last resort, we have the option of disabling SELinux entirely by editing /etc/selinux/config (all we did is change it to permissive mode, which prevents it from enforcing it’s ruleset), and then rebooting.

But, rebooting production servers sucks, so a couple of other thoughts before we get to that point –

What are the contents of the following files:

/etc/cron.deny
/etc/cron.allow
/etc/security/access.conf
/etc/pam.d/crond

It’s possible something in one of those will stand out as the culprit.
-Eric

ipsa · June 5, 2009, 12:36pm

/etc/cron.deny is empty
/etc/cron.allow doesn’t exist
/etc/security/access.conf only contains commented lines
Here are the contents of /etc/pam.d/crond:


auth	   sufficient pam_rootok.so
auth       required   pam_env.so
auth       include    system-auth
account    required   pam_access.so
account    include    system-auth
session    required   pam_loginuid.so
session    include    system-auth

Hopefully something here will help avoid a reboot.

Thanks!

Eric · June 5, 2009, 3:27pm

Yeah, none of those looks like a culprit.

What does this command output:

yum check-update