redhat-cluster-gfs2

Setting up a Hosting Environment, Part 3: RedHat Cluster and GFS2


Previous posts in this series:

Part 1: Setting up the servers

Part 2: Connecting the Array

RedHat Cluster and GFS2 Setup

Set date/time to be accurate and within a few minutes of each other.

  • Install the ntp program and update to current time.
    • yum install ntp
    • ntpdate time.nist.gov
  • Set time servers and start ntpd
    • service ntpd start
    • Edit the /etc/ntp.conf file to use the following servers:
  • Restart ntpd
    • service ntpd restart
    • chkconfig ntpd on

Cluster setup

RedHat Cluster must be set up before the GFS2 File systems can be created and mounted.

  • Instal the necessary programs
    • yum install openais cman rgmanager lvm2-cluster gfs2-utils ccs
    • Create a /etc/cluster/cluster.conf REMEMBER: Always increment the “config_version” parameter in the cluster tag!
    • We’ll be adding more to this later, but this will work for now.
    • Validate the config file
      • ccs_config_validate
    • Set a password for the ricci user
      • passwd ricci
    • Start ricci, and set to start on boot
      • service ricci start
      • chkconfig ricci on
    • Start modclusterd and set to start on boot
      • service modclusterd start
      • chkconfig modclusterd on
    • Sync the cluster.conf file to other node
      • ccs -f /etc/cluster/cluster.conf -h ted --setconf
    • Start cman on both servers at the same time
      • service cman start
    • Set cman to start on boot
      • chkconfig cman on
  • Check the tutorial on testing the fencing

Create GFS2 partitions

Create a partition on the new scsi device /dev/mapper/mpatha using parted. NOTE: This part only needs to be done once on one server

  • parted /dev/mapper/mpatha
  • mklabel gpt
  • mkpart primary 1 -1
  • set 1 lvm on
  • quit
  • Now you can see a partition for the storage array.
    • parted -l

Edit the /etc/lvm/lvm.conf file and set the value for locking_type = 3 to allow for cluster locking.

In order to enable the LVM volumes you are creating in a cluster, the cluster infrastructure must be running and the cluster must be quorate.

  • service clvmd start
  • chkconfig clvmd on
  • chkconfig gfs2 on

Create LVM partitions on the raw drive available from the StorageTek. NOTE: This part only needs to be done once on one server.

  • pvcreate /dev/mapper/mpatha1
  • vgcreate -c y StorageTek2530 /dev/mapper/mpatha1

Now create the different partitions for the system: sites, databases, logs, home, root

  • lvcreate --name sites --size 350GB StorageTek2530
  • lvcreate --name databases --size 100GB StorageTek2530
  • lvcreate --name logs --size 50GB StorageTek2530
  • lvcreate --name root --size 50GB StorageTek2530

Make a temporary directory /root-b and copy everything from root’s home directory to there, because it will be erased when we make the GFS2 file system.

Copy /root/.ssh/known_hosts to /etc/ssh/root_known_hosts so the file is different for both servers.

Before doing the home directory, we have to remove it from the local LVM.

  • unmount /home
  • lvremove bill_local/home and on ted lvremove ted_local/home
  • Remove the line from /etc/fstab referring to the /home directory on the local LVM
  • Then add the clustered LV.
    • lvcreate --name home --size 50GB StorageTek2530

Create GFS2 files systems on the LVM partitions created on the StorageTek. Make sure they are unmounted, first. NOTE: This part only needs to be done once on one server.

  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:sites /dev/mapper/StorageTek2530-sites
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:databases /dev/mapper/StorageTek2530-databases
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:logs /dev/mapper/StorageTek2530-logs
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:root /dev/mapper/StorageTek2530-root
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:home /dev/mapper/StorageTek2530-home

Mount the GFS2 partitions

  • NOTE: GFS2 file systems that have been mounted manually rather than automatically through an entry in the fstab file will not be known to the system when file systems are unmounted at system shutdown. As a result, the GFS2 script will not unmount the GFS2 file system. After the GFS2 shutdown script is run, the standard shutdown process kills off all remaining user processes, including the cluster infrastructure, and tries to unmount the file system. This unmount will fail without the cluster infrastructure and the system will hang.
  • To prevent the system from hanging when the GFS2 file systems are unmounted, you should do one of the following:
    • Always use an entry in the fstab file to mount the GFS2 file system.
    • If a GFS2 file system has been mounted manually with the mount command, be sure to unmount the file system manually with the umount command before rebooting or shutting down the system.
  • If your file system hangs while it is being unmounted during system shutdown under these circumstances, perform a hardware reboot. It is unlikely that any data will be lost since the file system is synced earlier in the shutdown process.

Make the appropriate folders on each node (/home is already there).

  • mkdir /sites /logs /databases

Make sure the appropriate lines are in /etc/fstab

Once the GFS2 partitions are set up and in /etc/fstab, rgmanager can be started. This will mount the GFS2 partions.

  • service rgmanager start
  • chkconfig rgmanager on

Starting Cluster Software

To start the cluster software on a node, type the following commands in this order:

  • service cman start
  • service clvmd start
  • service gfs2 start
  • service rgmanager start

Stopping Cluster Software

To stop the cluster software on a node, type the following commands in this order:

  • service ossec-hids stop
    • (ossec monitors the apache log files, so the /logs partition will not be unmounted unless ossec is stopped first.)
  • service rgmanager stop
  • service gfs2 stop
  • umount -at gfs2
  • service clvmd stop
  • service cman stop

Cluster tips

If a service shows as ‘failed’ when checking on services with clustat

  • Disable the service first: clusvcadm -d service-name
  • Then re-enable it: clusvcadm -e service-name

Have Shorewall start sooner in the boot process.

  • It was necessary to move shorewall up in the boot process, otherwise cman had no open connection to detect the other nodes.
  • Edit the /etc/init.d/shorewall and change the line near the top from # chkconfig: - 28 90 to
    • # chkconfig: - 18 90
  • Then use chkconfig to turn off shorewall and then back on.
    • chkconfig shorewall off
    • chkconfig shorewall on
Share and Enjoy:
  • Print
  • PDF
  • RSS

Related Posts:


5 thoughts on “Setting up a Hosting Environment, Part 3: RedHat Cluster and GFS2

  1. Pingback: Local machine trying to enable service:GFS...Failure

  2. kolmis

    pvcreate /dev/mapper/mpatha1
    vgcreate -c y StorageTek2530 /dev/mpatha1

    Should it be the same drive in both cases?

    Reply
    1. ammon Post author

      That is a typo, yes, they should be the same drive.

      should be:

      pvcreate /dev/mapper/mpatha1
      vgcreate -c y StorageTek2530 /dev/mapper/mpatha1

      I fixed that in the post. Thanks!

      Reply
  3. Evgen

    Why do you use rgmanager and services in cluster.conf?
    When one node is die and cluster have quorum, on other node gfs2 mount point works correctly without having to work rgmanager.

    Reply
    1. ammon Post author

      Good question. GFS2 is not the only thing managed by rgmanager. It also manages MySQL, Apache httpdand other resources shared by the cluster. If one node is running Apache httpd and fails, rgmanager is what kicks it over to the other node.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">