Setting up a Hosting Environment, Part 5: Apache and PHP

Figuring out the possibilities for Apache and PHP reminds me of a Dr. Seuss book, “Fox in Sox”. It’s a favorite of mine. I love reading it to the kids. In it, Mr. Fox tries to get Mr. Knox to say all kinds of ridiculous (in meaning and hard to say) tongue twisters. At one point Mr. Knox exclaims:
“I can’t blab such blibber blubber!
My tongue isn’t make of rubber.”

That’s what my brain felt like after trying to figure all of the options for Apache and PHP. To combat my rubber brain, I created this flow-chart to help me keep track of the options, the pros and cons for each, and the path I finally chose.

First off, a list of requirements and goals:

  1. Chroot each vhost to it’s own directory, and have Apache and PHP run on that vhost’s server account
  2. Speed, run Apache and PHP at their most effective and efficient levels
  3. Utilize an opcode cache, APC, to speed up PHP pages
  4. Use trusted repositories to make installation and upgrading easier

Here’s what I eventually figured out about Apache and PHP:

ApachePHP
Click on the image to see a larger view

These sites were helpful for the initial set up of PHP as CGI with mod_fcgi and Apache in chroot (mod_fcgi sends one request to each PHP process regardless if PHP children are available to handle more, and no sharing of APC opcode cache across PHP processes):

This site was helpful for setting up PHP as CGI with mod_fastcgi and Apache in chroot (mod_fastcgi sends multiple requests to a PHP process, so the process can send them to children processes, and having one PHP process for each site allows for APC opcode cache to be usable.)

These sites helped me learn about php-fpm and how it is not quite ready for what I have in mind:

I ended up going with Apache’s mod_fastcgi for using PHP as a CGI, and NOT using PHP-FPM, while running Apache in threaded mode with apache.worker enabled.

To get this set up is pretty easy. I already had Apache and PHP installed and running (with PHP as CGI using mod_fcgi), so here are the steps I used to convert it to run mod_fastcgi and apache.worker. I’m running CentOS 6.3.

Install the RPMForge repo for installing mod_fastcgi.

  • Get latest from http://repoforge.org/use/ : rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
  • yum --enablerepo=rpmforge install mod_fastcgi

Edit the /etc/httpd/conf/httpd.conf file

  • ServerTokens Prod
  • KeepAlive On
  • Edit the worker section. I still need to do some testing to figure out the best configuration
    <IfModule worker.c>
        StartServers         8
        MaxClients         300
        MinSpareThreads     25
        MaxSpareThreads     75
        ThreadsPerChild     25
        MaxRequestsPerChild  0
    </IfModule>
  • If there, make sure to comment out, or delete the lines for mod_php: LoadModule php5_module modules/libphp5.so
  • this line also: AddType application/x-httpd-php .php
  • The last line should be: Include conf/virtual_hosts.conf

 

Create a /etc/httpd/conf/virtual_hosts.conf file

Each virtual host needs to have an entry similar to this in the httpd.conf file, or I like to create a separate virtual_host.conf and include that in the main httpd.conf.

# Name-based virtual hosts
#

# Default
NameVirtualHost *:80

# Begin domain-name.com section
<VirtualHost *:80>
    DocumentRoot /var/domain-name/home/html/
    ServerName domain-name.com
    ServerAlias www.domain-name.com

    # Rewrite domain name to not use the 'www'
    RewriteEngine On
    RewriteCond %{HTTP_HOST}    !^domain-name\.com$ [NC]
    RewriteRule ^/(.*)  http://domain-name.com/$1 [R=301]

    # Specify where the error logs go for each domain
    ErrorLog /var/logs/httpd/current/domain-name.com-error_log
    CustomLog /var/logs/httpd/current/domain-name.com-access_log combined

    <IfModule mod_fastcgi.c>
        SuexecUserGroup domain-name domain-name
        ScriptAlias /cgi-bin/ "/var/www/cgi-bin/domain-name/"
        <Directory "/var/domain-name/home/html">
            Options -Indexes FollowSymLinks +ExecCGI
            AddHandler php5-fastcgi .php
            Action php5-fastcgi /cgi-bin/php-fastcgi
            Order allow,deny
            Allow from all
        </Directory>
    </IfModule>
</VirtualHost>
# End domain-name.com section

Things to note:

  • The line with SuexecUserGroup should have the user/group for the project.

Create the php-fastcgi file

Add a /var/www/cgi-bin/projectname/php-fastcgi file for each project. This allows php to run as FastCGI, and use suEXEC. The php-fastcgi file needs to be under suexec’s default directory path /var/www/cgi-bin/.

  • #!/bin/bash
    #  Set PHPRC to the path for the php.ini file. Change this to
    #  /var/projectname/home/ to let projects have their own php.ini file
    PHPRC=/var/domain-name/home/
    export PHPRC
    export PHP_FCGI_MAX_REQUESTS=5000
    export PHP_FCGI_CHILDREN=5
    exec /usr/bin/php-cgi

Things to note:

  • The directory and file created above must be have user/group of the project (the same as the user/group of the /var/projectname/ directory)
  • The directory and file must be executable and writable by the owner ONLY.
  • If you get Apache Internal Server errors, check /var/log/httpd/suexec.log
  • For each site, you can specify how much RAM the APC module can use. For large, busy sites, you set this higher. Not setting this defaults to 64MB, which is a bit more than needed for the average WP site. Change the last line in the /var/www/cgi-bin/projectname/php-fastcgi file:
    • exec /usr/bin/php-cgi -d apc.shm_size=128M

Change php.conf

Comment out everything in the /etc/httpd/conf.d/php.conf file so php is not loaded as a module when Apache starts.

Apache multi-threaded

Edit the /etc/sysconfig/httpd file to allow Apache to use multi-threaded mode (httpd.worker) which handles basic HTML files much nicer (less RAM). Uncomment the line with HTTPD=/usr/sbin/httpd.worker

Config Check

Check the Apache configuration files to see if there are any errors.

  • service httpd configtest

If all good, restart Apache

  • service httpd restart This will stop the running httpd service, and then start it again. Use this command after installing or removing a dynamically loaded module such as PHP. OR
  • service httpd reload This will cause the running httpd service to reload the configuration file. Note that any requests being currently processed will be interrupted, which may cause a client browser to display an error message or render a partial page. OR
  • service httpd graceful This will cause the running httpd service to reload the configuration file. Note that any requests being currently processed will use the old configuration.

Install APC

  • pecl install apc

Set up log rotation for Apache

  • Add a file /etc/logrotate.d/httpd.monti
  • /var/logs/httpd/*log {
        daily
        rotate 365
        compress
        missingok
        notifempty
        copytruncate
        olddir /var/logs/httpd/archives/
        sharedscripts
        postrotate
            /bin/kill -HUP `cat /var/run/httpd/httpd.pid 2>/dev/null` 2> /dev/null || true
        endscript
    }

Setting up a Hosting Environment, Part 3: RedHat Cluster and GFS2

Previous posts in this series:

Part 1: Setting up the servers

Part 2: Connecting the Array

RedHat Cluster and GFS2 Setup

Set date/time to be accurate and within a few minutes of each other.

  • Install the ntp program and update to current time.
    • yum install ntp
    • ntpdate time.nist.gov
  • Set time servers and start ntpd
    • service ntpd start
    • Edit the /etc/ntp.conf file to use the following servers:
    • server 0.pool.ntp.org
      server 1.pool.ntp.org
      server 2.pool.ntp.org
      server 3.pool.ntp.org
  • Restart ntpd
    • service ntpd restart
    • chkconfig ntpd on

Cluster setup

RedHat Cluster must be set up before the GFS2 File systems can be created and mounted.

  • Instal the necessary programs
    • yum install openais cman rgmanager lvm2-cluster gfs2-utils ccs
    • Create a /etc/cluster/cluster.conf REMEMBER: Always increment the “config_version” parameter in the cluster tag!
      • <?xml version=“1.0”?>
            <cluster config_version=“24” name=“web-production”>
                <cman expected_votes=“1” two_node=“1”/>
                <fence_daemon clean_start=“1” post_fail_delay=“6” post_join_delay=“3”/>
                <totem rrp_mode=“none” secauth=“off”/>
                <clusternodes>
                    <clusternode name=“bill” nodeid="1">
                        <fence>
                            <method name="ipmi">
                                <device action=“reboot” name=“ipmi_bill”/>
                            </method>
                        </fence>
                    </clusternode>
                    <clusternode name=“ted” nodeid="2">
                        <fence>
                            <method name="ipmi">
                                <device action=“reboot” name=“ipmi_ted”/>
                            </method>
                        </fence>
                    </clusternode>
                </clusternodes>
                <fencedevices>
                    <fencedevice agent=“fence_ipmilan” ipaddr=“billsp” login=“root” name=“ipmi_bill” passwd=“PASSWORD-HERE”/>
                    <fencedevice agent=“fence_ipmilan” ipaddr=“tedsp” login=“root” name=“ipmi_ted” passwd=“PASSWORD-HERE”/>
                </fencedevices>
                <rm log_level="5">
                    <resources>
                        <clusterfs device=“/dev/mapper/StorageTek2530-sites” fstype=“gfs2” mountpoint=“/sites” name=“sites”/>
                        <clusterfs device=“/dev/mapper/StorageTek2530-databases” fstype=“gfs2” mountpoint=“/databases” name=“databases”/>
                        <clusterfs device=“/dev/mapper/StorageTek2530-logs” fstype=“gfs2” mountpoint=“/logs” name=“logs”/>
                    </resources>
                    <failoverdomains>
                        <failoverdomain name=“bill-only” nofailback=“1” ordered=“0” restricted="1">
                            <failoverdomainnode name=“bill”/>
                        </failoverdomain>
                        <failoverdomain name=“ted-only” nofailback=“1” ordered=“0” restricted="1">
                            <failoverdomainnode name=“ted”/>
                        </failoverdomain>
                    </failoverdomains>
                </rm>
            </cluster>
    • We’ll be adding more to this later, but this will work for now.
    • Validate the config file
      • ccs_config_validate
    • Set a password for the ricci user
      • passwd ricci
    • Start ricci, and set to start on boot
      • service ricci start
      • chkconfig ricci on
    • Start modclusterd and set to start on boot
      • service modclusterd start
      • chkconfig modclusterd on
    • Sync the cluster.conf file to other node
      • ccs -f /etc/cluster/cluster.conf -h ted --setconf
    • Start cman on both servers at the same time
      • service cman start
    • Set cman to start on boot
      • chkconfig cman on
  • Check the tutorial on testing the fencing

Create GFS2 partitions

Create a partition on the new scsi device /dev/mapper/mpatha using parted. NOTE: This part only needs to be done once on one server

  • parted /dev/mapper/mpatha
  • mklabel gpt
  • mkpart primary 1 -1
  • set 1 lvm on
  • quit
  • Now you can see a partition for the storage array.
    • parted -l

Edit the /etc/lvm/lvm.conf file and set the value for locking_type = 3 to allow for cluster locking.

In order to enable the LVM volumes you are creating in a cluster, the cluster infrastructure must be running and the cluster must be quorate.

  • service clvmd start
  • chkconfig clvmd on
  • chkconfig gfs2 on

Create LVM partitions on the raw drive available from the StorageTek. NOTE: This part only needs to be done once on one server.

  • pvcreate /dev/mapper/mpatha1
  • vgcreate -c y StorageTek2530 /dev/mapper/mpatha1

Now create the different partitions for the system: sites, databases, logs, home, root

  • lvcreate --name sites --size 350GB StorageTek2530
  • lvcreate --name databases --size 100GB StorageTek2530
  • lvcreate --name logs --size 50GB StorageTek2530
  • lvcreate --name root --size 50GB StorageTek2530

Make a temporary directory /root-b and copy everything from root’s home directory to there, because it will be erased when we make the GFS2 file system.

Copy /root/.ssh/known_hosts to /etc/ssh/root_known_hosts so the file is different for both servers.

Before doing the home directory, we have to remove it from the local LVM.

  • unmount /home
  • lvremove bill_local/home and on ted lvremove ted_local/home
  • Remove the line from /etc/fstab referring to the /home directory on the local LVM
  • Then add the clustered LV.
    • lvcreate --name home --size 50GB StorageTek2530

Create GFS2 files systems on the LVM partitions created on the StorageTek. Make sure they are unmounted, first. NOTE: This part only needs to be done once on one server.

  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:sites /dev/mapper/StorageTek2530-sites
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:databases /dev/mapper/StorageTek2530-databases
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:logs /dev/mapper/StorageTek2530-logs
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:root /dev/mapper/StorageTek2530-root
  • mkfs.gfs2 -p lock_dlm -j 2 -t web-production:home /dev/mapper/StorageTek2530-home

Mount the GFS2 partitions

  • NOTE: GFS2 file systems that have been mounted manually rather than automatically through an entry in the fstab file will not be known to the system when file systems are unmounted at system shutdown. As a result, the GFS2 script will not unmount the GFS2 file system. After the GFS2 shutdown script is run, the standard shutdown process kills off all remaining user processes, including the cluster infrastructure, and tries to unmount the file system. This unmount will fail without the cluster infrastructure and the system will hang.
  • To prevent the system from hanging when the GFS2 file systems are unmounted, you should do one of the following:
    • Always use an entry in the fstab file to mount the GFS2 file system.
    • If a GFS2 file system has been mounted manually with the mount command, be sure to unmount the file system manually with the umount command before rebooting or shutting down the system.
  • If your file system hangs while it is being unmounted during system shutdown under these circumstances, perform a hardware reboot. It is unlikely that any data will be lost since the file system is synced earlier in the shutdown process.

Make the appropriate folders on each node (/home is already there).

  • mkdir /sites /logs /databases

Make sure the appropriate lines are in /etc/fstab

#GFS2 partitions shared in the cluster
/dev/mapper/StorageTek2530-root        /root        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-home        /home        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-databases      /databases      gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-logs        /logs        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-sites    /sites    gfs2   defaults,acl    0 0

Once the GFS2 partitions are set up and in /etc/fstab, rgmanager can be started. This will mount the GFS2 partions.

  • service rgmanager start
  • chkconfig rgmanager on

Starting Cluster Software

To start the cluster software on a node, type the following commands in this order:

  • service cman start
  • service clvmd start
  • service gfs2 start
  • service rgmanager start

Stopping Cluster Software

To stop the cluster software on a node, type the following commands in this order:

  • service ossec-hids stop
    • (ossec monitors the apache log files, so the /logs partition will not be unmounted unless ossec is stopped first.)
  • service rgmanager stop
  • service gfs2 stop
  • umount -at gfs2
  • service clvmd stop
  • service cman stop

Cluster tips

If a service shows as ‘failed’ when checking on services with clustat

  • Disable the service first: clusvcadm -d service-name
  • Then re-enable it: clusvcadm -e service-name

Have Shorewall start sooner in the boot process.

  • It was necessary to move shorewall up in the boot process, otherwise cman had no open connection to detect the other nodes.
  • Edit the /etc/init.d/shorewall and change the line near the top from # chkconfig: - 28 90 to
    • # chkconfig: - 18 90
  • Then use chkconfig to turn off shorewall and then back on.
    • chkconfig shorewall off
    • chkconfig shorewall on

Setting up a Hosting Environment – Part 2: Connecting the Storage Array

[See Part 1: The Servers]

One of the most frustrating parts of this set up was getting the storage array talking to the servers. I finally got it figured out. I’m using a StorageTek 2530 to connect to two SunFire X2100 M2’s via SAS (Serial Attached SCSI) cables. I put in a dual port SAS HBA (Host Bus Adapter) in the X2100 M2’s, but for real redundancy, I should have used two single port HBA’s. The Sun/Oracle documentation is pretty good about how to physically set up the servers and storage array, but are pretty lacking from there on.

StorageTek 2530 Set Up

Replace the parts in squares brackets below with whatever you want.

  • Install the Sun CAM software.
    • Grab the latest version from http://support.oracle.com
      • You’ll need an active support contract and have an account.
      • Go to the ‘Patches and Updates’ tab.
      • Click on the ‘Product or Family (Advanced)’ link
      • In the ‘Product is’ section start typing in ‘Sun Storage Common Array Manager (CAM)’ and select it from the list
      • In the ‘Release is’ section select the most recent version
      • For the last section, select ‘Platform’ and then select ‘Linux x86-64’
      • Click ‘Search’
      • Click the ‘Download’ link for the software.
      • Upload the tar file to the server.
    • Pre-requisite software that needs to be installed.
      • yum install ksh bc /lib/ld-linux.so.2 libgcc.i686 libstdc++.i686 libzip.i686 gettext
    • Once CAM software is downloaded, un-zipped, un-tarred or what have you, change directories to HostSoftwareCD_6.9.0.16/components and install the jdk available there:
      • rpm -Uvh jdk-6u20-linux-i586.rpm
    • Next run the RunMe.bin file in the HostSoftwareCD_6.9.0.16 folder
      • ./RunMe.bin -c
    • Agree to all License Agreement stuffs
    • Select the Typical install.
  • Add the /opt/sun/cam/binfolder to path
    • With root using tcsh add this to .tcshrc
      • setenv PATH ${PATH}:/opt/sun/cam/bin
    • Then do source .tcshrc
  • Make sure there is an IP on the same subnet as the array (192.168.128.xxx)
    • Make a /etc/sysconfig/network-scripts/ifcfg-eth1:1file and put this in there
      • DEVICE=“eth1:1”
        BOOTPROTO=static
        HWADDR=“xx:xx:xx:xx:xx:xx:xx”
        IPADDR=192.168.128.xxx
        NM_CONTROLLED=“no”
        ONBOOT=“yes”
    • Install the RAID Proxy Agent package located in the Add_On/RaidArrayProxy directory of the latest CAM software distribution. (I found this to be optional.)
      • rpm -ivh SMruntime.xx.xx.xx.xx-xxxx.rpm
      • rpm -ivh SMagent-LINUX-xx.xx.xx.xx-xxxx.rpm
  • Register the StorageTek with the host. Process can take several minutes.
    • sscs register -d storage-system
  • Once registered, you can name the array anything you want. Note what the array is named from the previous step.
  • sscs modify -T [Array-Name] array ARRAY1
  • Set up the storage profile, pool, disk, volume, mapping. Use the command line commands below, or set it up via the web interface. NOTE: This part only needs to be done on one of the hosts.
    • If using the web interface, you have to use a windows laptop hooked up to the local network (192.168.128.xxx), or perhaps a server in the same local network that is not running CentOS 6, which has a known issue where the web interface does not work. For the web interface connect to https://localhost:6789 using the laptop or server Administrator/root account information.
    • sscs create -a knox pool [Pool-Name]
    • sscs create -a knox -p [Pool-Name] -n 11 vdisk [Vdisk-Name]
    • sscs create -a knox -p [Pool-Name] -s max -v [Vdisk-Name] volume [Volume-Name]
  • Create the host group and apply to host.
    • sscs create -a knox hostgroup [ApacheHosts]
  • Create hosts and assign to hostgroup
    • sscs create -a knox -g [ApacheHosts] host [Host-Name] and repeat for other hosts.
  • Map volume to host group
    • sscs map -a knox -g ApacheHosts volume Volume-Name
  • The array volume should now be available as /dev/sdb and /dev/sdc because the hosts are connected by two SAS cables each.
  • It took me a while to grasp the meaning for the different terms: pool, volume, volume groups, disks, etc. I drew up a chart with the appropriate commands to create the different aspects.

    To utilize both cables connecting the server to the storage array, the OS needs to use multi-pathing. I had lots of troubles trying to set this up after the OS was installed, so I just let it be done by the installer. Here’s what should happen if you find the OS already installed and need to set up multi-paths.

    • Set up DM-Multipath
      • NOTE: This is taken care of during the OS installation.
      • Multipath allows both SAS connections to the storage array to appear as one connection to the server. This allows for data to pass through even if one cable suddenly stops working, it seamlessly fails to the other path. For example, taken the image above, if the connection between hba1->cntrlr1 goes down, you still have connection hba2->cntrlr2. The OS sees one connection, and just uses whichever path is working.
      • After Multipath is set up, the storage array will be available as a device at /dev/mapper/mpatha. This will be the device to partition, format, and throw LVM on.
      • Install the multipath program and dependents
        • yum install device-mapper-multipath
      • Run mpathconf --enable to create a default /etc/multipath.conffile or create one using the following:
        • #  multipath.conf written by anacondadefaults {
          user_friendly_names yes
          }
          blacklist {
          devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
          devnode “^hd[a-z]”
          devnode “^dcssblk[0-9]*”
          device {
          vendor “DGC”
          product “LUNZ”
          }
          device {
          vendor “IBM”
          product “S/390.*”
          }
          # don’t count normal SATA devices as multipaths
          device {
          vendor “ATA”
          }
          # don’t count 3ware devices as multipaths
          device {
          vendor “3ware”
          }
          device {
          vendor “AMCC”
          }
          # nor highpoint devices
          device {
          vendor “HPT”
          }
          wwid “3600508e000000000c9c1189277b84b05”
          device {
          vendor TEAC
          product DV-28E-V
          }
          wwid “*”
          }
          blacklist_exceptions {
          wwid “3600a0b80003abca4000007284f33c167”
          }
          multipaths {
          multipath {
          uid 0
          gid 0
          wwid “3600a0b80003abca4000007284f33c167”
          mode 0600
          }
          }
      • Set multipathd to start on boot, and if not on, turn it on
        • chkconfig multipathd on
        • service multipathd start

    Setting up a Hosting Environment: Part 1 – The servers

    I’ve spent a lot of time at work setting up a few servers to be our new production environment. Much of it was accomplished by reading the documentation over and over again. Not much out there on the Net, so I’m hoping this series of posts benefits someone else out there.

    First of all, I’ll cover what set up I would like to achieve and why.

    Hardware

    I’m using two Sun SunFire X2100 M2 connected to a StorageTek 2530 with 4.5TB of drive space. The servers attach to the storage array via SCSI cables for quick data transfer speeds. The array also has the ability to handle iSCSI connections. This will give me a decent base set up, with room to grow.

    Set up

    I’ll put the two servers in a cluster and make the services available over the cluster. They will share the storage using GFS2. In the future, I’ll add a couple of load balancer/proxy machines to farm out the Web traffic, and add a couple more SunFire X2100 M2’s to take that load. One of the main reasons to set up a new configuration with new servers is to provide a clean environment for the many WordPress and Omeka installations we host. We’ve had to hang on to some legacy services to support some older projects, so this will allow us to keep up to date. It will also allow me to set up Apache and PHP to run as a server user, locked down to it’s own directory. That way each of the 100+ sites won’t be able to access any other site’s content. I picked CentOS as the OS because it has cluster and GFS2 options of RedHat, but without the cost.

    Sun X2100 M2 OS Install steps

    1. Boot up with CentOS 6.x Minimal Install CD for x86_64
    2. Select the option to ‘Install or upgrade an existing system’, then hit the Enter key
    3. Skip the media test.
    4. You are now in graphic install mode.
    5. Hit Enter for ‘OK’ for ’English as the language.
    6. Hit Enter for ‘OK’ to US keyboard.
    7. Select the option to do a “Specialized Storage Devices” install
    8. Enter the computer name ‘bill.com’ or ‘ted.com’, etc
    9. Click the button to ‘Configure Network’.
      1. Eth2 seems to be the one associated with port 0 on the servers, so select that one and then ‘Add’
      2. Select ‘Connect Automatically’.
      3. Click the ‘IPv4 Settings’ tab.
      4. Choose ‘Manual’ for the ‘Method’.
      5. Enter the following for the info in ‘Addresses’.
        1. Address: 192.168.1.1
        2. Netmask: 255.255.255.0
        3. Gateway: 192.168.1.1
      6. For ‘DNS servers’, enter 192.168.1.100
      7. Then ‘Apply’
    10. Select ‘Next’ to keep the defaults for time zone and system clock.
    11. Enter a root password
    12. DRIVE PARTITION SETUP
      1. On the ‘Basic Devices’ tab, select the local drive and on the ‘Multipath Devices’ tab, select the storage array, and click ‘Next’.
      2. Select the ‘Fresh Installation’ option for a fresh install, or ‘Upgrade an Existing Installation’ to upgrade. Hit ‘Next’.
      3. Select ‘Create custom layout.’ and ‘Next’
      4. Delete all of the current LVM and other partitions.
      5. Select the free remaining drive for the local drive (should be /dev/sda). Click ‘Create’
      6. BOOT PARTITION
        1. Select ‘Standard Partition’ and click ‘Create’
        2. Set the Mount Point as /boot, the File System Type as ‘ext4’ and the Size (MB) as 500, then click ‘OK’
      7. Select the free space and click ‘Create’
      8. LVM PARTITION(NOTE: The sizes are different based on the size of the hard drives.)
        1. Select ‘LVM Physical Volume’ and click ‘Create’
        2. Select ‘Fill to maximum allowable size’ and click ‘OK’
        3. Select the new LVM partition and click ‘Create’
        4. Select ‘LVM Volume Group’ and click ‘Create’
        5. Set the ‘Volume Group Name’ as ‘Local’  then click the ‘Add’ button
        6. Set the ‘File System Type’ as swap, the ‘Logical Volume Name’ as ‘swap’ and the ‘Size(MB)’ as ‘12288’, then click ‘OK’.
        7. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘root’, and the ‘Size(MB)’ to ‘51200’. Then click ‘OK’.
        8. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/home’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘home’, and the ‘Size(MB)’ to ‘500’. Then click ‘OK’.
        9. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/var’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘var’, and the ‘Size(MB)’ to the remaining space available. Then click ‘OK’.
        10. Click ‘OK’
      9. Click ‘Next’ and ‘Write changes to disk’ to finish the partition creation.
    13. Leave the boot loader settings as is, and click ‘Next’
    14. Select the ‘Minimal’ option and click ‘Next’

    One of the most important things to have with servers is some form of remote management. That way you don’t need to trek down to the data center each time the server hangs while testing (and it happens a lot). For Sun systems, that means setting up the ELOM (Embedded Lights Out Manager).

    Steps to set up the Remote Console (Embedded Lights Out Manager – ELOM) for SunFire X2100 M2

    Set the SP serial port rate to 115200.

    • Log into the web based console, or through ssh via a computer on the same subnet (https://192.168.1.10) The IP is whatever the IP is set for the ELOM device. Check in BIOS for the IP.
      • Go to the Configuration tab, then the Serial Port tab.
      • Change the Baud Rate to 115200.

    Set BIOS

    IPMI Config
       Set LAN Config
       Set PEF Config
         PEF Support ........ [Enabled]
         PEF Action Global
            All of them ..... [Enabled]
         Alert Startup Discover ..... [Disabled]
         Startup Delay .............. [Disabled]
         Event Message For PEF ...... [Disabled]
       BMC Watch Dog Timer Action ... [Disabled]
       External Com Port ............ [BMC]
    Remote Access
       Remote Access ................ [Serial]
       Serial Port Number ........... [Com2]
       Serial Port Mode ............. [115200 8,n,1]
       Flow Control ................. [Hardware]
       Post-Boot Support ............ [Always]
       Terminal Type ................ [VT100]
       VT-UTF8 Combo Key ............ [Enabled]
    • Other options for the Serial Port Mode are 9600, 19200, 38400, and 57600

    Edit Linux Config Files

    Add a /etc/init/serial-ttyS1.conf file

    RedHat in EL 6, and thereby CentOS, moved to Upstart instead of Sysv, so we create a new serial-ttyS1.conf file instead of editing the /etc/inittab file.

    #  This service maintains a getty on /dev/ttyS1.
    stop on runlevel [016]
    
    respawn
    instance $TTY
    exec /sbin/mingetty $TTY

    Change grub.conf

    # grub.conf generated by anaconda
    #
    # Note that you do not have to rerun grub after making changes to this file
    # NOTICE:  You have a /boot partition.  This means that
    #          all kernel and initrd paths are relative to /boot/, eg.
    #          root (hd0,0)
    #          kernel /vmlinuz-version ro root=/dev/Logical/root
    #          initrd /initrd-version.img
    #boot=/dev/sda
    default=0
    timeout=5
    #splashimage=(hd0,0)/grub/splash.xpm.gz
    #hiddenmenu
    serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
    terminal --timeout=10 serial console
    
    title CentOS Linux (2.6.32-71.29.1.el6.x86_64)
            root (hd0,0)
            kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/Local-root \
    rd_LVM_LV=Local/root rd_LVM_LV=Local/swap rd_NO_LUKS rd_NO_MD rd_NO_DM \
    console=tty1 console=ttyS1,115200n8
              initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img

    Add line to securetty

    console
    vc/1
    vc/2
    vc/3
    vc/4
    vc/5
    vc/6
    vc/7
    vc/8
    vc/9
    vc/10
    vc/11
    tty1
    tty2
    tty3
    tty4
    tty5
    tty6
    tty7
    tty8
    tty9
    tty10
    tty11
    ttyS1

    SUN SP Commands

    Connect to the ELOM by ssh into the IP address.
    ssh root@192.168.xxx.xxx

    • To power on the host, enter the following command:
      • set /SP/SystemInfo/CtrlInfo PowerCtrl=on
    • To power off the host gracefully, enter the following command:
      • set /SP/SystemInfo/CtrlInfo PowerCtrl=gracefuloff
    • To power off the host forcefully, enter the following command:
      • set /SP/SystemInfo/CtrlInfo PowerCtrl=forceoff
    • To reset the host, enter the following command:
      • set /SP/SystemInfo/CtrlInfo PowerCtrl=reset
    • To reboot and enter the BIOS automatically, enter the following command:
      • set /SP/SystemInfo/CtrlInfo BootCtrl=BIOSSetup
    • To change the IP address for the ELOM, enter:
      • set /SP/AgentInfo IpAddress=xxx.xxx.xxx.xxx
    • The default user name is root, and the default password is changeme.
      • set /SP/User/[username] Password=[password]
    • To start a session on the server console, enter this command:
      • start /SP/AgentInfo/console
      • To revert to CLI once the console has been started, press Esc-Shift-9 keys.
    • 
To terminate a server console session started by another user, enter this command:
      • stop /SP/AgentInfo/console

    Next we secure the new servers with some software updates and a firewall.

    Software Updates and installs:

    1. Edit /etc/resolve.conf
    2. nameserver 192.168.1.100
      options single-request-reopen

    3. yum install openssh-clients tcsh ksh bc rpm-build gcc gcc-c++ redhat-rpm-config acl gcc gnupg make vim-enhanced man wget which mlocate bzip2-devel libxml2-devel screen sudo parted gd-devel pam_passwdqc.x86_64 rsync zip xorg-x11-server-utils gettext
    4. disable SELinux. Edit the /etc/sysconfig/selinux file and set SELINUX=disabled.
      • Change takes affect on next reboot.
    5. Add the following lines to the /etc/vimrcfile:
      set autoindent ” auto indent after {
      set smartindent ” same
      set shiftwidth=4 ” number of space characters inserted for indentation
      set expandtab ” inserts spaces instead of tabs
      set tabstop=4 ” number of spaces the tab is.
      set pastetoggle=<C-P> ” Ctrl-P toggles paste mode
    6. Switch root shell to tcsh
      • Edit the /etc/passwdfile to have root use tcshroot:x:0:0:root:/root:/bin/tcsh
      • Edit the .tcshrcfile in root’s home.
        #  .tcshrc#  User specific aliases and functionsalias rm ‘rm -i’
        alias cp ‘cp -i’
        alias mv ‘mv -i’set prompt='[%n@%m %c]# ‘

        setenv PATH ${PATH}:/opt/sun/cam/bin

        #  Make command completion (TAB key) cycle through all possible choices
        #  (The default is to simply display a list of all choices when more than one
        #  match is available.)
        bindkey “^I” complete-word-fwd

      • Logout and back in for it to take affect.
    7. Edit /etc/hosts. Add a line with IP and domain name.
      #  Do not remove the following line, or various programs
      #  that require network functionality will fail.
      127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6#  External IPs
      192.168.1.1 bill.com
      192.168.1.2 ted.com192.168.1.3 domain.com # this needs to be an IP that the cluster server can manage#  Internal IPs
      192.168.1.11 bill.localdomain bill # notice the .localdomain, this is necessary for mysql later
      192.168.1.12 ted.localdomain ted othernode # this is bill’s hosts file. othernode would be on the bill line for ted’s hosts file.
      #  ServicePort IPs
      192.168.1.21 billsp # I like to have a short name to use to connect to the service port (ELOM)
      192.168.1.22 tedsp

      #  Internal Services
      192.168.1.100 http.localdomain httpd.localdomain
      192.168.1.101 mysql.localdomain
      192.168.1.102 memcached.localdomain

    8. Run updatedb to set up the locate database.
    9. Edit password settings to allow for stricter control over passwords. This requires strong passwords or the use of passphrases.
    10. [Optional] Firefox: yum update, and then ayum install firefox xorg-x11-xauth xorg-x11-fonts-Type1There will be more you’ll need too.
      • If you get this error: process 702: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/var/lib/dbus/machine-id": No such file or directory. Then run the following command as root.
        • dbus-uuidgen > /var/lib/dbus/machine-id
    11. Set up ssh keys
      • ssh-keygen
      • Copy the id_rsa.pub file to the other node
      • Copy the contents of id_rsa.pub to cat id_rsa.pub >> ~/.ssh/authorized_keys
      • Double check permission on authorized_keys and id_rsa both set to rw-------
      • You should now be able to log in from bill to ted (and vice versa) without a password.
    
    

    Shorewall

    • Yum Install:
      • Get EPEL repository. Visit http://fedoraproject.org/wiki/EPEL to get the URL for the correct rpm. Something like: epel-release-6-5.noarch.rpm.
      • Copy that URL and runrpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpmon the machine.
      • Edit the /etc/yum.repos.d/epel.repo file and set the first “enabled” line to equal 0. That disables yum from using the EPEL repo by default.
      • Install shorewall with yum.yum --enablerepo=epel install shorewall
    • Enable program to run by editing the /etc/shorewall/shorewall.conf file. Change the STARTUP_ENABLED=NOtoSTARTUP_ENABLED=Yes
    • Edit the shorewall config files.
    • Edit the /etc/shorewall/zonesfile:
      • #
        #  Shorewall version 4 – Zones File
        #
        #  For information about this file, type “man shorewall-zones”
        #
        #  The manpage is also online at
        #  http://www.shorewall.net/manpages/shorewall-zones.html
        #
        ###############################################################################
        #ZONE TYPE OPTIONS IN OUT
        #  OPTIONS OPTIONSnet ipv4 # The big bad Internet
        loc ipv4 # Internal LAN
        fw firewall#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE
    • Edit the /etc/shorewall/interfacesfile:
      • #
        #  Shorewall version 4 – Interfaces File
        #
        #  For information about entries in this file, type “man shorewall-interfaces”
        #
        #  The manpage is also online at
        #  http://www.shorewall.net/manpages/shorewall-interfaces.html
        #
        ###############################################################################
        #ZONE INTERFACE BROADCAST OPTIONS
        net eth2
        loc eth1
    • Edit the /etc/shorewall/policyfile:
      • ###############################################################################
        #SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
        #  LEVEL BURST MASK
        #  To/from internal lan
        fw loc ACCEPT
        loc fw ACCEPT
        #  To/from net
        fw net ACCEPT
        net all DROP info
        #
        #  THE FOLLOWING POLICY MUST BE LAST
        #
        all all REJECT info
        #LAST LINE — DO NOT REMOVE
    • Edit the /etc/shorewall/rulesfile:
      • ######################################################################################
        #ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL
        #  PORT PORT DEST
        #SECTION ESTABLISHED
        #SECTION RELATED
        SECTION NEWSECTION NEW#  Standard services
        #
        ACCEPT  net      fw      tcp     ssh
        ACCEPT  net      fw      tcp     80,443Ping/ACCEPT      net      fw

        #LAST LINE — ADD YOUR ENTRIES BEFORE THIS ONE — DO NOT REMOVE

    • Edit the /etc/shorewall/routestoppedfile:
      • #
        #  Shorewall version 4 – Routestopped File
        #
        #  For information about entries in this file, type “man shorewall-routestopped”
        #
        #  The manpage is also online at
        #  http://www.shorewall.net/manpages/shorewall-routestopped.html
        #
        #  See http://shorewall.net/starting_and_stopping_shorewall.htm for additional
        #  information.
        #
        ###############################################################################
        #INTERFACE HOST OPTIONS PROTO DEST SOURCE
        PORT PORT
        eth1     –
        eth2     –
    • Set shorewall to start on reboots.chkconfig shorewall on
    • Start shorewall:service shorewall start

    The next part will be connecting the servers to the storage array.