Two SunFire X2100 M2 servers connected to a StorageTek 2530 via iscsi. The two nodes are running CentOS 6.2 with RedHat cluster software. I have a server running Nagios for monitoring and it checks for failed disks on the StorageTek by running a script on either node that returns the number of “optimal” and “failed” disks.
Updating system software is important. Keeping packages up to date protects from security vulnerabilities. Unfortunately, sometimes it breaks things. In this case, updating the suggested packages broke my set up, making it so that the Sun Storage CAM (Common Array Manager) software did not work anymore.
I first became alerted to this when Nagios sent me errors from the script checking the StorageTek disks. I checked the command that runs in the script to see what was up, and it returned several errors. Here they are for future googlers:
sscs list -i 192.168.128.101 device
returned “Command failed due to an exception. null” and
sscs list -a arrayname host
returned “arrayname : The resource was not found.”
Not particularly helpful messages.
Fortunately the nodes could still mount the array partitions, which allowed them to continue running as web and mysql servers. I just couldn’t run management commands on the array.
Since some of the required software for CAM was updated, I supposed that was causing the issue. The required software is listed below:
I couldn’t figure out on my own exactly what was wrong, so I contacted Oracle support, and they finally tipped me off to the solution. Completely remove the CAM software and reinstall it. Those steps are outlined below:
/var/opt/CommonArrayManager/Host_Software_6.9.0.16/bin/
and run./uninstall -f
yum update
and restart the server if needed.components
in there. Change into that directory and install the jdk available there:rpm -Uvh jdk-6u20-linux-i586.rpm
RunMe.bin
file in the CAM Software CD folder../RunMe.bin -c
rpm -ivh SMruntime.xx.xx.xx.xx-xxxx.rpm
rpm -ivh SMagent-LINUX-xx.xx.xx.xx-xxxx.rpm
sscs register -d storage-system
One additional issue I ran into, was that some update or other process shutdown the NIC connecting the node to the array. I had to make sure that was running before I ran the register -d storage-system command above.
]]>If you have a live production MySQL server, stopping it to make a backup is not really an option. Fortunately there are a few options. Before you decide on which option to choose, here is a list of things to keep in mind when choosing a backup solution (from the MySQL gurus at Percona):
There are a few MySQL backup products out there as well. I have used the first two on this list.
There’s probably a gazillion more out there. Google’s your friend in finding things you need.
There are several options. You could use a script above, or create a slave of the database (basically an exact copy of the production MySQL server – all changes that occur in the master are sent to the slave), or some combination. I’ll use a combination. I’ll replicate the production server onto the backup server, then run the incremental backups from there. This first part will walk through the process of setting up MySQL replication.
To give proper credit, here are several other how to’s I found helpful.
Step 1. Edit the my.cnf file to include at least the following (if needed) lines. Note: you will have to restart MySQL for these changes to take affect.
[mysqld] server_id=1 innodb_flush_log_at_trx_commit=1 log_bin=mysql-bin.log sync_binlog=1
Step 2. Make a MySQL user for the slave to use.
GRANT REPLICATION SLAVE ON *.* TO 'rep_user'@'localhost' IDENTIFIED BY 'passwordhere';
Step 3. Open a terminal session and log in to a MySQL prompt. Type the following command and hit enter.
FLUSH TABLES WITH READ LOCK
Note: This will lock your database so that no changes can be made from any web applications or other programs. This session should remain open, and the database locked for the next few steps.
Step 4. After the FLUSH TABLES command finishes, run the following command and press enter.
SHOW MASTER STATUS
Record the information under “File Name” and “Position”.
Step 5. Make a copy of the database files.
5.1 LVM Snapshot:
In another terminal session, run the following command to make an LVM snapshot of the database.
lvcreate -L10G -s -n mysql-backup /dev/mapper/dbases
This creates a snapshot of the database files very quickly. We can use the snapshot later to copy the data to the backup server without interfering with the original database files.
After this command finishes, you can unlock the database as shown in the next step. Then you can mount the new LVM partition and copy the files to the backup server.
mkdir -p /mnt/mysql-backup
mount -o nouuid /dev/mapper/mysql-backup /mnt/mysql-backup
rsync -avz -e "ssh -c blowfish" /mnt/mysql-backup [email protected]:/backup/location
5.2 RSYNC:
If you don’t have your database files on an LVM partition, you can just copy the files to the backup server now using rsync, scp or what have you. This will take significantly longer (depending on the size of your database), leaving the database in a locked state.
rsync -avz -e "ssh -c blowfish" /dbases/mysql [email protected]:/backup/location
5.3 MySQL Dump:
You could also take a mysqldump of the database and copy that SQL file to the other server.
mysqldump -uuser -p --all-databases > mysql-backup.sql
scp mysql-backup.sql [email protected]:/backup/location
Step 6. Once the lvcreate command has finished, you can unlock the database.
UNLOCK TABLES
Step 7. If you haven’t already, copy the copy of the database files to the backup server.
Step 1. Edit the my.cnf file to include at least the following (if needed) lines. Note: you will have to restart MySQL for these changes to take affect.
[mysqld] server_id=2
Step 2. Start MySQL and run the following commands in a mysql session to start the MySQL slave.
CHANGE MASTER TO MASTER_HOST = "master.server.com", MASTER_USER = "rep_user", MASTER_PASSWORD = "passwordhere", MASTER_LOG_FILE = "mysql-bin.log", MASTER_LOG_POS = 2341234;
The MASTER_HOST is the domain name or IP address of the master server. MASTER_USER, MASTER_PASSWORD were created on the master server in Step 2. MASTER_LOG_FILE and MASTER_LOG_POS were gathered in Step 4.Then, finally, to start the slave, issue the following command in mysql.
START SLAVE;