mySQL not starting since kernel update

singh90 · September 22, 2013, 6:31pm

Hi guys,

I posted on Experts-Exchange but havent had the fix that i require yet.

So I’ve been an idiot and upgraded the kernel on my virtualmin server… everything works smoothly except for mysql

no matter what i try, i am unable to bring the service back to life.

sudo service mysql start
start: Job failed to start

uname -a
Linux hostname-removed 3.2.0-53-generic #81-Ubuntu SMP Thu Aug 22 21:01:03 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I am running Ubuntu 12.04 and Virtualmin 4.02 - I have several sites set up which depend on mysql. Each site worked perfectly until the upgrade!

Any ideas?

Locutus · September 22, 2013, 7:49pm

Firstly, you’re certainly not an idiot for updating the kernel. It’s suggested to do so from time to time.

Second, if all else fails, you can always revert back to the previous kernel. You even get a grub boot menu where you can select it, since the automatic process keeps all the previous kernels.

Then, to know why MySQL fails to start you might want to check /var/log/syslog for error messages. Without further information about that, it’s hard to say what could be the reason. If there’s nothing in the syslog, start mysqld from the command line and check its output.

singh90 · September 22, 2013, 8:49pm

unfortunately to add complication this is a virtually hosted system so i cant access grub. unless this can be done from cmd line or VirtualMin CP?

and even further complication, im a bit of a noob

I have tried to start mysqld but it fails to start - the service is in a stop/waiting status and cant be stopped manually either.

Locutus · September 23, 2013, 2:08am

Please give us more details about “does not start”, what exactly did you try and what messages do you get? What’s in the syslog?

To select kernel during boot, you need console access. Does your virtual host provider offer that? Otherwise you can uninstall the new kernel’s package using aptitude. But to be honest, I have doubts that the new kernel is the problem. New kernels rarely have such specific problems that one service doesn’t work anymore but the rest does.

singh90 · September 23, 2013, 1:24pm

unfortunately I don’t have console access

This is what I have tried so far.


/usr/bin/mysqld_safe --mysqld
130911 22:09:16 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
130911 22:09:16 mysqld_safe Logging to '/var/log/mysql/error.log'.
130911 22:09:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130911 22:09:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

error.log
130911 20:39:11 [Note] /usr/sbin/mysqld: Shutdown complete

130911 22:09:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130911 22:09:16 [Note] Plugin ‘FEDERATED’ is disabled.
130911 22:09:16 InnoDB: The InnoDB memory heap is disabled
130911 22:09:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130911 22:09:16 InnoDB: Compressed tables use zlib 1.2.3.4
a/usr/sbin/mysqld: Can’t create/write to file ‘/tmp/ibzUySNF’ (Errcode: 13)
130911 22:09:16 InnoDB: Error: unable to create temporary file; errno: 13
130911 22:09:16 [ERROR] Plugin ‘InnoDB’ init function returned error.
130911 22:09:16 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
130911 22:09:16 [ERROR] /usr/sbin/mysqld: unknown option ‘–mysqld’
130911 22:09:16 [ERROR] Aborting

130911 22:09:16 [Note] /usr/sbin/mysqld: Shutdown complete

130911 22:09:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

my.cnf

The MySQL database server configuration file.

You can copy this to one of:

- “/etc/mysql/my.cnf” to set global options,

- “~/.my.cnf” to set user-specific options.

One can use all long options that the program supports.

Run program with --help to get a list of available options and with

–print-defaults to see which it would actually understand and use.

For explanations see

http://dev.mysql.com/doc/mysql/en/server-system-variables.html

This will be passed to all mysql clients

It has been reported that passwords should be enclosed with ticks/quotes

escpecially if they contain “#” chars…

Remember to edit /etc/mysql/debian.cnf when changing the socket location.

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

Here is entries for some specific programs

The following values assume you have at least 32M ram

This was formally known as [safe_mysqld]. Both versions are currently parsed.

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]

* Basic Settings

user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

Instead of skip-networking the default is now to listen only on

localhost which is more compatible and is not less secure.

bind-address = 127.0.0.1

* Fine Tuning

key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8

This replaces the startup script and checks MyISAM tables if needed

the first time they are touched

myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10

* Query Cache Configuration

query_cache_limit = 1M
query_cache_size = 16M

* Logging and Replication

Both location gets rotated by the cronjob.

Be aware that this log type is a performance killer.

As of 5.1 you can enable the log at runtime!

#general_log_file = /var/log/mysql/mysql.log
#general_log = 1

Error log - should be very few entries.

log_error = /var/log/mysql/error.log

Here you can see queries with especially long duration

#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes

The following can be used as easy to replay backup logs or for replication.

note: if you are setting up a replication slave, see README.Debian about

other settings you may need to change.

#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name

* InnoDB

InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.

Read the manual for more InnoDB related options. There are many!

* Security Features

Read the manual, too, if you want chroot!

chroot = /var/lib/mysql/

For generating SSL certificates I recommend the OpenSSL GUI “tinyca”.

ssl-ca=/etc/mysql/cacert.pem

ssl-cert=/etc/mysql/server-cert.pem

ssl-key=/etc/mysql/server-key.pem

[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

* IMPORTANT: Additional settings that can override those from this file!

The files must end with ‘.cnf’, otherwise they’ll be ignored.

!includedir /etc/mysql/conf.d/

MySQL
#!/bin/sh -e

upstart-job

Symlink target for initscripts that have been converted to Upstart.

set -e

INITSCRIPT="$(basename “$0”)"
JOB="${INITSCRIPT%.sh}"

if [ “$JOB” = “upstart-job” ]; then
if [ -z “$1” ]; then
echo “Usage: upstart-job JOB COMMAND” 1>&2
exit 1
fi

JOB="$1"
INITSCRIPT="$1"
shift

else
if [ -z “$1” ]; then
echo “Usage: $0 COMMAND” 1>&2
exit 1
fi
fi

COMMAND="$1"
shift

if [ -z “$DPKG_MAINTSCRIPT_PACKAGE” ]; then
ECHO=echo
else
ECHO=:
fi

$ECHO “Rather than invoking init scripts through /etc/init.d, use the service(8)”
$ECHO “utility, e.g. service $INITSCRIPT $COMMAND”

Only check if jobs are disabled if the currently running version of

Upstart (which may be older than the latest installed version)

supports such a query.

This check is necessary to handle the scenario when upgrading from a

release without the ‘show-config’ command (introduced in

Upstart for Ubuntu version 0.9.7) since without this check, all

installed packages with associated Upstart jobs would be considered

disabled.

Once Upstart can maintain state on re-exec, this change can be

dropped (since the currently running version of Upstart will always

match the latest installed version).

UPSTART_VERSION_RUNNING=$(initctl version|awk ‘{print $3}’|tr -d ‘)’)

if dpkg --compare-versions “$UPSTART_VERSION_RUNNING” ge 0.9.7
then
initctl show-config -e “$JOB”|grep -q ‘^ start on’ || DISABLED=1
fi

case $COMMAND in
status)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
$COMMAND “$JOB”
;;
start|stop)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
if status “$JOB” 2>/dev/null | grep -q ’ start/’; then
RUNNING=1
fi
if [ -z “$RUNNING” ] && [ “$COMMAND” = “stop” ]; then
exit 0
elif [ -n “$RUNNING” ] && [ “$COMMAND” = “start” ]; then
exit 0
elif [ -n “$DISABLED” ] && [ “$COMMAND” = “start” ]; then
exit 0
fi
$COMMAND “$JOB”
;;
restart)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the stop(8) and then start(8) utilities,”
$ECHO “e.g. stop $JOB ; start $JOB. The restart(8) utility is also available.”
if status “$JOB” 2>/dev/null | grep -q ’ start/’; then
RUNNING=1
fi
if [ -n “$RUNNING” ] ; then
stop “$JOB”
fi
# If the job is disabled and is not currently running, the job is
# not restarted. However, if the job is disabled but has been forced into the
# running state, we do stop and restart it since this is expected behaviour
# for the admin who forced the start.
if [ -n “$DISABLED” ] && [ -z “$RUNNING” ]; then
exit 0
fi
start “$JOB”
;;
reload|force-reload)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the reload(8) utility, e.g. reload $JOB”
reload “$JOB”
;;
*)
$ECHO
$ECHO “The script you are attempting to invoke has been converted to an Upstart” 1>&2
$ECHO “job, but $COMMAND is not supported for Upstart jobs.” 1>&2
exit 1
esac


service mysql stop
stop: Unknown instance:
 service mysql start
start: Job failed to start


 ps -ef | grep -i mysql
root      6450  6394  0 22:05 pts/0    00:00:00 grep --color=auto -i mysql


grep -i mysql /var/log/syslog
Sep 12 20:48:07 mtcweb001 kernel: [977751.708986] type=1400 audit(1379015287.718:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8707 comm="apparmor_parser"
Sep 12 20:48:07 mtcweb001 kernel: [977751.759375] init: mysql main process (8711) terminated with status 1
Sep 12 20:48:07 mtcweb001 kernel: [977751.759434] init: mysql main process ended, respawning
Sep 12 20:48:08 mtcweb001 kernel: [977752.759492] init: mysql post-start process (8712) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.778580] type=1400 audit(1379015288.786:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8735 comm="apparmor_parser"
Sep 12 20:48:08 mtcweb001 kernel: [977752.829382] init: mysql main process (8739) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.829467] init: mysql main process ended, respawning
Sep 12 20:48:09 mtcweb001 kernel: [977753.828191] init: mysql post-start process (8740) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.849473] type=1400 audit(1379015289.857:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8763 comm="apparmor_parser"
Sep 12 20:48:09 mtcweb001 kernel: [977753.905021] init: mysql main process (8767) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.905077] init: mysql respawning too fast, stopped
Sep 12 20:50:59 mtcweb001 kernel: [977923.035704] type=1400 audit(1379015459.024:13): apparmor="STATUS" operation="profile_load" name="/usr/sbin/mysqld" pid=953 comm="apparmor_parser"
Sep 12 20:51:00 mtcweb001 kernel: [977924.623310] type=1400 audit(1379015460.612:19): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1026 comm="apparmor_parser"
Sep 12 20:51:05 mtcweb001 kernel: [977929.498249] init: mysql main process (1030) terminated with status 1
Sep 12 20:51:05 mtcweb001 kernel: [977929.498306] init: mysql main process ended, respawning
Sep 12 20:51:06 mtcweb001 kernel: [977930.437181] init: mysql post-start process (1032) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.458488] type=1400 audit(1379015466.447:20): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1242 comm="apparmor_parser"
Sep 12 20:51:06 mtcweb001 kernel: [977930.507946] init: mysql main process (1246) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.508000] init: mysql main process ended, respawning
Sep 12 20:51:07 mtcweb001 kernel: [977931.511377] init: mysql post-start process (1247) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.532272] type=1400 audit(1379015467.519:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1287 comm="apparmor_parser"
Sep 12 20:51:07 mtcweb001 kernel: [977931.583652] init: mysql main process (1291) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.583723] init: mysql respawning too fast, stopped
Sep 12 20:53:53 mtcweb001 kernel: [978097.688889] type=1400 audit(1379015633.652:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2280 comm="apparmor_parser"
Sep 12 20:53:53 mtcweb001 kernel: [978097.738975] init: mysql main process (2284) terminated with status 1
Sep 12 20:53:53 mtcweb001 kernel: [978097.739045] init: mysql main process ended, respawning
Sep 12 20:53:54 mtcweb001 kernel: [978098.748114] init: mysql post-start process (2285) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.767137] type=1400 audit(1379015634.728:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2314 comm="apparmor_parser"
Sep 12 20:53:54 mtcweb001 kernel: [978098.818450] init: mysql main process (2318) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.818507] init: mysql main process ended, respawning
Sep 12 20:53:55 mtcweb001 kernel: [978099.817846] init: mysql post-start process (2319) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.837249] type=1400 audit(1379015635.800:24): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2342 comm="apparmor_parser"
Sep 12 20:53:55 mtcweb001 kernel: [978099.884365] init: mysql main process (2346) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.884430] init: mysql respawning too fast, stopped
Sep 12 20:57:27 mtcweb001 kernel: [978311.560296] type=1400 audit(1379015847.499:25): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2861 comm="apparmor_parser"
Sep 12 20:57:27 mtcweb001 kernel: [978311.717120] init: mysql main process (2865) terminated with status 1
Sep 12 20:57:27 mtcweb001 kernel: [978311.717178] init: mysql main process ended, respawning
Sep 12 20:57:28 mtcweb001 kernel: [978312.612552] init: mysql post-start process (2866) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.632940] type=1400 audit(1379015848.571:26): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2896 comm="apparmor_parser"
Sep 12 20:57:28 mtcweb001 kernel: [978312.680481] init: mysql main process (2900) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.680589] init: mysql main process ended, respawning
Sep 12 20:57:29 mtcweb001 kernel: [978313.685133] init: mysql post-start process (2901) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.704668] type=1400 audit(1379015849.643:27): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2930 comm="apparmor_parser"
Sep 12 20:57:29 mtcweb001 kernel: [978313.761914] init: mysql main process (2934) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.761971] init: mysql respawning too fast, stopped
Sep 12 20:59:51 mtcweb001 kernel: [978455.908352] type=1400 audit(1379015991.823:28): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3027 comm="apparmor_parser"
Sep 12 20:59:51 mtcweb001 kernel: [978455.958808] init: mysql main process (3031) terminated with status 1
Sep 12 20:59:51 mtcweb001 kernel: [978455.958864] init: mysql main process ended, respawning
Sep 12 20:59:52 mtcweb001 kernel: [978456.963349] init: mysql post-start process (3032) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978456.982963] type=1400 audit(1379015992.899:29): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3063 comm="apparmor_parser"
Sep 12 20:59:52 mtcweb001 kernel: [978457.033235] init: mysql main process (3067) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978457.033291] init: mysql main process ended, respawning
Sep 12 20:59:53 mtcweb001 kernel: [978458.034843] init: mysql post-start process (3068) terminated with status 1
Sep 12 20:59:53 mtcweb001 kernel: [978458.054455] type=1400 audit(1379015993.971:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3100 comm="apparmor_parser"
Sep 12 20:59:54 mtcweb001 kernel: [978458.105610] init: mysql main process (3104) terminated with status 1
Sep 12 20:59:54 mtcweb001 kernel: [978458.105678] init: mysql respawning too fast, stopped
root@mtcweb001:~# clear
root@mtcweb001:~# grep -i mysql /var/log/syslog
Sep 12 20:48:07 mtcweb001 kernel: [977751.708986] type=1400 audit(1379015287.718:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8707 comm="apparmor_parser"
Sep 12 20:48:07 mtcweb001 kernel: [977751.759375] init: mysql main process (8711) terminated with status 1
Sep 12 20:48:07 mtcweb001 kernel: [977751.759434] init: mysql main process ended, respawning
Sep 12 20:48:08 mtcweb001 kernel: [977752.759492] init: mysql post-start process (8712) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.778580] type=1400 audit(1379015288.786:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8735 comm="apparmor_parser"
Sep 12 20:48:08 mtcweb001 kernel: [977752.829382] init: mysql main process (8739) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.829467] init: mysql main process ended, respawning
Sep 12 20:48:09 mtcweb001 kernel: [977753.828191] init: mysql post-start process (8740) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.849473] type=1400 audit(1379015289.857:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8763 comm="apparmor_parser"
Sep 12 20:48:09 mtcweb001 kernel: [977753.905021] init: mysql main process (8767) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.905077] init: mysql respawning too fast, stopped
Sep 12 20:50:59 mtcweb001 kernel: [977923.035704] type=1400 audit(1379015459.024:13): apparmor="STATUS" operation="profile_load" name="/usr/sbin/mysqld" pid=953 comm="apparmor_parser"
Sep 12 20:51:00 mtcweb001 kernel: [977924.623310] type=1400 audit(1379015460.612:19): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1026 comm="apparmor_parser"
Sep 12 20:51:05 mtcweb001 kernel: [977929.498249] init: mysql main process (1030) terminated with status 1
Sep 12 20:51:05 mtcweb001 kernel: [977929.498306] init: mysql main process ended, respawning
Sep 12 20:51:06 mtcweb001 kernel: [977930.437181] init: mysql post-start process (1032) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.458488] type=1400 audit(1379015466.447:20): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1242 comm="apparmor_parser"
Sep 12 20:51:06 mtcweb001 kernel: [977930.507946] init: mysql main process (1246) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.508000] init: mysql main process ended, respawning
Sep 12 20:51:07 mtcweb001 kernel: [977931.511377] init: mysql post-start process (1247) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.532272] type=1400 audit(1379015467.519:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1287 comm="apparmor_parser"
Sep 12 20:51:07 mtcweb001 kernel: [977931.583652] init: mysql main process (1291) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.583723] init: mysql respawning too fast, stopped
Sep 12 20:53:53 mtcweb001 kernel: [978097.688889] type=1400 audit(1379015633.652:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2280 comm="apparmor_parser"
Sep 12 20:53:53 mtcweb001 kernel: [978097.738975] init: mysql main process (2284) terminated with status 1
Sep 12 20:53:53 mtcweb001 kernel: [978097.739045] init: mysql main process ended, respawning
Sep 12 20:53:54 mtcweb001 kernel: [978098.748114] init: mysql post-start process (2285) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.767137] type=1400 audit(1379015634.728:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2314 comm="apparmor_parser"
Sep 12 20:53:54 mtcweb001 kernel: [978098.818450] init: mysql main process (2318) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.818507] init: mysql main process ended, respawning
Sep 12 20:53:55 mtcweb001 kernel: [978099.817846] init: mysql post-start process (2319) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.837249] type=1400 audit(1379015635.800:24): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2342 comm="apparmor_parser"
Sep 12 20:53:55 mtcweb001 kernel: [978099.884365] init: mysql main process (2346) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.884430] init: mysql respawning too fast, stopped
Sep 12 20:57:27 mtcweb001 kernel: [978311.560296] type=1400 audit(1379015847.499:25): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2861 comm="apparmor_parser"
Sep 12 20:57:27 mtcweb001 kernel: [978311.717120] init: mysql main process (2865) terminated with status 1
Sep 12 20:57:27 mtcweb001 kernel: [978311.717178] init: mysql main process ended, respawning
Sep 12 20:57:28 mtcweb001 kernel: [978312.612552] init: mysql post-start process (2866) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.632940] type=1400 audit(1379015848.571:26): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2896 comm="apparmor_parser"
Sep 12 20:57:28 mtcweb001 kernel: [978312.680481] init: mysql main process (2900) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.680589] init: mysql main process ended, respawning
Sep 12 20:57:29 mtcweb001 kernel: [978313.685133] init: mysql post-start process (2901) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.704668] type=1400 audit(1379015849.643:27): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2930 comm="apparmor_parser"
Sep 12 20:57:29 mtcweb001 kernel: [978313.761914] init: mysql main process (2934) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.761971] init: mysql respawning too fast, stopped
Sep 12 20:59:51 mtcweb001 kernel: [978455.908352] type=1400 audit(1379015991.823:28): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3027 comm="apparmor_parser"
Sep 12 20:59:51 mtcweb001 kernel: [978455.958808] init: mysql main process (3031) terminated with status 1
Sep 12 20:59:51 mtcweb001 kernel: [978455.958864] init: mysql main process ended, respawning
Sep 12 20:59:52 mtcweb001 kernel: [978456.963349] init: mysql post-start process (3032) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978456.982963] type=1400 audit(1379015992.899:29): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3063 comm="apparmor_parser"
Sep 12 20:59:52 mtcweb001 kernel: [978457.033235] init: mysql main process (3067) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978457.033291] init: mysql main process ended, respawning
Sep 12 20:59:53 mtcweb001 kernel: [978458.034843] init: mysql post-start process (3068) terminated with status 1
Sep 12 20:59:53 mtcweb001 kernel: [978458.054455] type=1400 audit(1379015993.971:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3100 comm="apparmor_parser"
Sep 12 20:59:54 mtcweb001 kernel: [978458.105610] init: mysql main process (3104) terminated with status 1
Sep 12 20:59:54 mtcweb001 kernel: [978458.105678] init: mysql respawning too fast, stopped

This is the troubleshooting I’ve carried out so far and posted on Experts-Exchange!

Hope this helps!

Locutus · September 23, 2013, 1:36pm

The key error seems to be this in your error.log:

a/usr/sbin/mysqld: Can't create/write to file '/tmp/ibzUySNF' (Errcode: 13) 130911 22:09:16 InnoDB: Error: unable to create temporary file; errno: 13

Make sure your /tmp directory has proper permissions and, if it is on a separate partition, is mounted without the noexec option.

root@australis:~# ls -ld /tmp drwxrwxrwt 9 root root 4096 Sep 23 15:28 /tmp root@australis:~# mount [...] /dev/sdd1 on /tmp type ext4 (rw,noatime)

singh90 · September 26, 2013, 6:32pm

sorry for the late reply… i thought i had subscriptions set!

can you tell me what permissions need to be set? i can connect via winscp and make the amendments or ssh if required

Also, what do you mean by mounted with noexec option?

singh90 · September 26, 2013, 7:01pm

just re-read your post - tmp is on the same partition.

It is also permissioned 1777

Locutus · September 26, 2013, 7:19pm

Okay, so directory permissions seem to be okay. Are you sure the kernel was the only thing that got upgraded when MySQL stopped working? Do you have SELinux or AppArmor installed? Those might prevent MySQL from writing to /tmp.

singh90 · September 26, 2013, 7:22pm

I’ve never installed SELinux or AppArmor - unless they come packaged with other tools?

Locutus · September 26, 2013, 8:01pm

Well Ubuntu by default installs AppArmor in the 10.x and 12. server versions… So you might have it without knowing it.

singh90 · September 26, 2013, 10:24pm

just removed it

but more worryingly im now getting this


root@mtcweb001:~# sudo service mysql start
start: Job failed to start
root@mtcweb001:~# sudo service mysql stop
stop: Unknown instance:

singh90 · September 26, 2013, 10:52pm

I did, i’ve tried everything that I’ve found so far

I read that permissions could be an issue like you suggested and even disabled logging in the cnf file to no avail.

Its looking like it might need to be a rebuild just worried that I dont have a backup of the databases!

Locutus · September 26, 2013, 10:53pm

That’s the same error you had before.

The only idea I have left right now is that you could try changing the MySQL temporary files path in my.cnf, change /tmp to e.g. /var/lib/mysql/tmp (after creating that directory with the proper owner and permissions). If MySQL has permission to write anywhere, then there.

Otherwise, did you try googling for the error messages you see? This “can’t create /tmp/blabla, error 13” thing?

Locutus · September 26, 2013, 10:54pm

I edited my previous post with the last idea I have for now.

singh90 · September 27, 2013, 9:20am

Oh wow. I feel like a really really big idiot!!!

I’ve been looking at /var/tmp

I just took a look at /tmp and saw perms were set to 755 so I changed it to 1777. Now MySQL starts up!

massive facepalm