Setting up a Hosting Environment: Part 1 – The servers


I’ve spent a lot of time at work setting up a few servers to be our new production environment. Much of it was accomplished by reading the documentation over and over again. Not much out there on the Net, so I’m hoping this series of posts benefits someone else out there.

First of all, I’ll cover what set up I would like to achieve and why.

Hardware

I’m using two Sun SunFire X2100 M2 connected to a StorageTek 2530 with 4.5TB of drive space. The servers attach to the storage array via SCSI cables for quick data transfer speeds. The array also has the ability to handle iSCSI connections. This will give me a decent base set up, with room to grow.

Set up

I’ll put the two servers in a cluster and make the services available over the cluster. They will share the storage using GFS2. In the future, I’ll add a couple of load balancer/proxy machines to farm out the Web traffic, and add a couple more SunFire X2100 M2’s to take that load. One of the main reasons to set up a new configuration with new servers is to provide a clean environment for the many WordPress and Omeka installations we host. We’ve had to hang on to some legacy services to support some older projects, so this will allow us to keep up to date. It will also allow me to set up Apache and PHP to run as a server user, locked down to it’s own directory. That way each of the 100+ sites won’t be able to access any other site’s content. I picked CentOS as the OS because it has cluster and GFS2 options of RedHat, but without the cost.

Sun X2100 M2 OS Install steps

  1. Boot up with CentOS 6.x Minimal Install CD for x86_64
  2. Select the option to ‘Install or upgrade an existing system’, then hit the Enter key
  3. Skip the media test.
  4. You are now in graphic install mode.
  5. Hit Enter for ‘OK’ for ’English as the language.
  6. Hit Enter for ‘OK’ to US keyboard.
  7. Select the option to do a “Specialized Storage Devices” install
  8. Enter the computer name ‘bill.com’ or ‘ted.com’, etc
  9. Click the button to ‘Configure Network’.
    1. Eth2 seems to be the one associated with port 0 on the servers, so select that one and then ‘Add’
    2. Select ‘Connect Automatically’.
    3. Click the ‘IPv4 Settings’ tab.
    4. Choose ‘Manual’ for the ‘Method’.
    5. Enter the following for the info in ‘Addresses’.
      1. Address: 192.168.1.1
      2. Netmask: 255.255.255.0
      3. Gateway: 192.168.1.1
    6. For ‘DNS servers’, enter 192.168.1.100
    7. Then ‘Apply’
  10. Select ‘Next’ to keep the defaults for time zone and system clock.
  11. Enter a root password
  12. DRIVE PARTITION SETUP
    1. On the ‘Basic Devices’ tab, select the local drive and on the ‘Multipath Devices’ tab, select the storage array, and click ‘Next’.
    2. Select the ‘Fresh Installation’ option for a fresh install, or ‘Upgrade an Existing Installation’ to upgrade. Hit ‘Next’.
    3. Select ‘Create custom layout.’ and ‘Next’
    4. Delete all of the current LVM and other partitions.
    5. Select the free remaining drive for the local drive (should be /dev/sda). Click ‘Create’
    6. BOOT PARTITION
      1. Select ‘Standard Partition’ and click ‘Create’
      2. Set the Mount Point as /boot, the File System Type as ‘ext4’ and the Size (MB) as 500, then click ‘OK’
    7. Select the free space and click ‘Create’
    8. LVM PARTITION(NOTE: The sizes are different based on the size of the hard drives.)
      1. Select ‘LVM Physical Volume’ and click ‘Create’
      2. Select ‘Fill to maximum allowable size’ and click ‘OK’
      3. Select the new LVM partition and click ‘Create’
      4. Select ‘LVM Volume Group’ and click ‘Create’
      5. Set the ‘Volume Group Name’ as ‘Local’  then click the ‘Add’ button
      6. Set the ‘File System Type’ as swap, the ‘Logical Volume Name’ as ‘swap’ and the ‘Size(MB)’ as ‘12288’, then click ‘OK’.
      7. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘root’, and the ‘Size(MB)’ to ‘51200’. Then click ‘OK’.
      8. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/home’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘home’, and the ‘Size(MB)’ to ‘500’. Then click ‘OK’.
      9. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/var’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘var’, and the ‘Size(MB)’ to the remaining space available. Then click ‘OK’.
      10. Click ‘OK’
    9. Click ‘Next’ and ‘Write changes to disk’ to finish the partition creation.
  13. Leave the boot loader settings as is, and click ‘Next’
  14. Select the ‘Minimal’ option and click ‘Next’

One of the most important things to have with servers is some form of remote management. That way you don’t need to trek down to the data center each time the server hangs while testing (and it happens a lot). For Sun systems, that means setting up the ELOM (Embedded Lights Out Manager).

Steps to set up the Remote Console (Embedded Lights Out Manager – ELOM) for SunFire X2100 M2

Set the SP serial port rate to 115200.

  • Log into the web based console, or through ssh via a computer on the same subnet (https://192.168.1.10) The IP is whatever the IP is set for the ELOM device. Check in BIOS for the IP.
    • Go to the Configuration tab, then the Serial Port tab.
    • Change the Baud Rate to 115200.

Set BIOS

  • Other options for the Serial Port Mode are 9600, 19200, 38400, and 57600

Edit Linux Config Files

Add a /etc/init/serial-ttyS1.conf file

RedHat in EL 6, and thereby CentOS, moved to Upstart instead of Sysv, so we create a new serial-ttyS1.conf file instead of editing the /etc/inittab file.

Change grub.conf

Add line to securetty


SUN SP Commands

Connect to the ELOM by ssh into the IP address.
ssh root@192.168.xxx.xxx

  • To power on the host, enter the following command:
    • set /SP/SystemInfo/CtrlInfo PowerCtrl=on
  • To power off the host gracefully, enter the following command:
    • set /SP/SystemInfo/CtrlInfo PowerCtrl=gracefuloff
  • To power off the host forcefully, enter the following command:
    • set /SP/SystemInfo/CtrlInfo PowerCtrl=forceoff
  • To reset the host, enter the following command:
    • set /SP/SystemInfo/CtrlInfo PowerCtrl=reset
  • To reboot and enter the BIOS automatically, enter the following command:
    • set /SP/SystemInfo/CtrlInfo BootCtrl=BIOSSetup
  • To change the IP address for the ELOM, enter:
    • set /SP/AgentInfo IpAddress=xxx.xxx.xxx.xxx
  • The default user name is root, and the default password is changeme.
    • set /SP/User/[username] Password=[password]
  • To start a session on the server console, enter this command:
    • start /SP/AgentInfo/console
    • To revert to CLI once the console has been started, press Esc-Shift-9 keys.
  • 
To terminate a server console session started by another user, enter this command:
    • stop /SP/AgentInfo/console

Next we secure the new servers with some software updates and a firewall.

Software Updates and installs:

  1. Edit /etc/resolve.conf
  2. nameserver 192.168.1.100
    options single-request-reopen

  3. yum install openssh-clients tcsh ksh bc rpm-build gcc gcc-c++ redhat-rpm-config acl gcc gnupg make vim-enhanced man wget which mlocate bzip2-devel libxml2-devel screen sudo parted gd-devel pam_passwdqc.x86_64 rsync zip xorg-x11-server-utils gettext
  4. disable SELinux. Edit the /etc/sysconfig/selinux file and set SELINUX=disabled.
    • Change takes affect on next reboot.
  5. Add the following lines to the /etc/vimrcfile:
    set autoindent ” auto indent after {
    set smartindent ” same
    set shiftwidth=4 ” number of space characters inserted for indentation
    set expandtab ” inserts spaces instead of tabs
    set tabstop=4 ” number of spaces the tab is.
    set pastetoggle=<C-P> ” Ctrl-P toggles paste mode
  6. Switch root shell to tcsh
    • Edit the /etc/passwdfile to have root use tcshroot:x:0:0:root:/root:/bin/tcsh
    • Edit the .tcshrcfile in root’s home.
      #  .tcshrc#  User specific aliases and functionsalias rm ‘rm -i’
      alias cp ‘cp -i’
      alias mv ‘mv -i’set prompt='[%n@%m %c]# ‘

      setenv PATH ${PATH}:/opt/sun/cam/bin

      #  Make command completion (TAB key) cycle through all possible choices
      #  (The default is to simply display a list of all choices when more than one
      #  match is available.)
      bindkey “^I” complete-word-fwd

    • Logout and back in for it to take affect.
  7. Edit /etc/hosts. Add a line with IP and domain name.
    #  Do not remove the following line, or various programs
    #  that require network functionality will fail.
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6#  External IPs
    192.168.1.1 bill.com
    192.168.1.2 ted.com192.168.1.3 domain.com # this needs to be an IP that the cluster server can manage#  Internal IPs
    192.168.1.11 bill.localdomain bill # notice the .localdomain, this is necessary for mysql later
    192.168.1.12 ted.localdomain ted othernode # this is bill’s hosts file. othernode would be on the bill line for ted’s hosts file.
    #  ServicePort IPs
    192.168.1.21 billsp # I like to have a short name to use to connect to the service port (ELOM)
    192.168.1.22 tedsp

    #  Internal Services
    192.168.1.100 http.localdomain httpd.localdomain
    192.168.1.101 mysql.localdomain
    192.168.1.102 memcached.localdomain

  8. Run updatedb to set up the locate database.
  9. Edit password settings to allow for stricter control over passwords. This requires strong passwords or the use of passphrases.
  10. [Optional] Firefox: yum update, and then ayum install firefox xorg-x11-xauth xorg-x11-fonts-Type1There will be more you’ll need too.
    • If you get this error: process 702: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/var/lib/dbus/machine-id": No such file or directory. Then run the following command as root.
      • dbus-uuidgen > /var/lib/dbus/machine-id
  11. Set up ssh keys
    • ssh-keygen
    • Copy the id_rsa.pub file to the other node
    • Copy the contents of id_rsa.pub to cat id_rsa.pub >> ~/.ssh/authorized_keys
    • Double check permission on authorized_keys and id_rsa both set to rw-------
    • You should now be able to log in from bill to ted (and vice versa) without a password.

Shorewall

  • Yum Install:
    • Get EPEL repository. Visit http://fedoraproject.org/wiki/EPEL to get the URL for the correct rpm. Something like: epel-release-6-5.noarch.rpm.
    • Copy that URL and runrpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpmon the machine.
    • Edit the /etc/yum.repos.d/epel.repo file and set the first “enabled” line to equal 0. That disables yum from using the EPEL repo by default.
    • Install shorewall with yum.yum --enablerepo=epel install shorewall
  • Enable program to run by editing the /etc/shorewall/shorewall.conf file. Change the STARTUP_ENABLED=NOtoSTARTUP_ENABLED=Yes
  • Edit the shorewall config files.
  • Edit the /etc/shorewall/zonesfile:
    • #
      #  Shorewall version 4 – Zones File
      #
      #  For information about this file, type “man shorewall-zones”
      #
      #  The manpage is also online at
      #  http://www.shorewall.net/manpages/shorewall-zones.html
      #
      ###############################################################################
      #ZONE TYPE OPTIONS IN OUT
      #  OPTIONS OPTIONSnet ipv4 # The big bad Internet
      loc ipv4 # Internal LAN
      fw firewall#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE
  • Edit the /etc/shorewall/interfacesfile:
    • #
      #  Shorewall version 4 – Interfaces File
      #
      #  For information about entries in this file, type “man shorewall-interfaces”
      #
      #  The manpage is also online at
      #  http://www.shorewall.net/manpages/shorewall-interfaces.html
      #
      ###############################################################################
      #ZONE INTERFACE BROADCAST OPTIONS
      net eth2
      loc eth1
  • Edit the /etc/shorewall/policyfile:
    • ###############################################################################
      #SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
      #  LEVEL BURST MASK
      #  To/from internal lan
      fw loc ACCEPT
      loc fw ACCEPT
      #  To/from net
      fw net ACCEPT
      net all DROP info
      #
      #  THE FOLLOWING POLICY MUST BE LAST
      #
      all all REJECT info
      #LAST LINE — DO NOT REMOVE
  • Edit the /etc/shorewall/rulesfile:
    • ######################################################################################
      #ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL
      #  PORT PORT DEST
      #SECTION ESTABLISHED
      #SECTION RELATED
      SECTION NEWSECTION NEW#  Standard services
      #
      ACCEPT  net      fw      tcp     ssh
      ACCEPT  net      fw      tcp     80,443Ping/ACCEPT      net      fw

      #LAST LINE — ADD YOUR ENTRIES BEFORE THIS ONE — DO NOT REMOVE

  • Edit the /etc/shorewall/routestoppedfile:
    • #
      #  Shorewall version 4 – Routestopped File
      #
      #  For information about entries in this file, type “man shorewall-routestopped”
      #
      #  The manpage is also online at
      #  http://www.shorewall.net/manpages/shorewall-routestopped.html
      #
      #  See http://shorewall.net/starting_and_stopping_shorewall.htm for additional
      #  information.
      #
      ###############################################################################
      #INTERFACE HOST OPTIONS PROTO DEST SOURCE
      PORT PORT
      eth1     –
      eth2     –
  • Set shorewall to start on reboots.chkconfig shorewall on
  • Start shorewall:service shorewall start

The next part will be connecting the servers to the storage array.

Share and Enjoy:
  • Print
  • PDF
  • RSS

Related Posts: