Setting up a Hosting Environment: Part 1 – The servers

I’ve spent a lot of time at work setting up a few servers to be our new production environment. Much of it was accomplished by reading the documentation over and over again. Not much out there on the Net, so I’m hoping this series of posts benefits someone else out there.

First of all, I’ll cover what set up I would like to achieve and why.

Hardware

I’m using two Sun SunFire X2100 M2 connected to a StorageTek 2530 with 4.5TB of drive space. The servers attach to the storage array via SCSI cables for quick data transfer speeds. The array also has the ability to handle iSCSI connections. This will give me a decent base set up, with room to grow.

Set up

I’ll put the two servers in a cluster and make the services available over the cluster. They will share the storage using GFS2. In the future, I’ll add a couple of load balancer/proxy machines to farm out the Web traffic, and add a couple more SunFire X2100 M2’s to take that load. One of the main reasons to set up a new configuration with new servers is to provide a clean environment for the many WordPress and Omeka installations we host. We’ve had to hang on to some legacy services to support some older projects, so this will allow us to keep up to date. It will also allow me to set up Apache and PHP to run as a server user, locked down to it’s own directory. That way each of the 100+ sites won’t be able to access any other site’s content. I picked CentOS as the OS because it has cluster and GFS2 options of RedHat, but without the cost.

Sun X2100 M2 OS Install steps

Boot up with CentOS 6.x Minimal Install CD for x86_64
Select the option to ‘Install or upgrade an existing system’, then hit the Enter key
Skip the media test.
You are now in graphic install mode.
Hit Enter for ‘OK’ for ’English as the language.
Hit Enter for ‘OK’ to US keyboard.
Select the option to do a “Specialized Storage Devices” install
Enter the computer name ‘bill.com’ or ‘ted.com’, etc
Click the button to ‘Configure Network’.
1. Eth2 seems to be the one associated with port 0 on the servers, so select that one and then ‘Add’
2. Select ‘Connect Automatically’.
3. Click the ‘IPv4 Settings’ tab.
4. Choose ‘Manual’ for the ‘Method’.
5. Enter the following for the info in ‘Addresses’.
  1. Address: 192.168.1.1
  2. Netmask: 255.255.255.0
  3. Gateway: 192.168.1.1
6. For ‘DNS servers’, enter 192.168.1.100
7. Then ‘Apply’
Select ‘Next’ to keep the defaults for time zone and system clock.
Enter a root password
DRIVE PARTITION SETUP
1. On the ‘Basic Devices’ tab, select the local drive and on the ‘Multipath Devices’ tab, select the storage array, and click ‘Next’.
2. Select the ‘Fresh Installation’ option for a fresh install, or ‘Upgrade an Existing Installation’ to upgrade. Hit ‘Next’.
3. Select ‘Create custom layout.’ and ‘Next’
4. Delete all of the current LVM and other partitions.
5. Select the free remaining drive for the local drive (should be /dev/sda). Click ‘Create’
6. BOOT PARTITION
  1. Select ‘Standard Partition’ and click ‘Create’
  2. Set the Mount Point as /boot, the File System Type as ‘ext4’ and the Size (MB) as 500, then click ‘OK’
7. Select the free space and click ‘Create’
8. LVM PARTITION(NOTE: The sizes are different based on the size of the hard drives.)
  1. Select ‘LVM Physical Volume’ and click ‘Create’
  2. Select ‘Fill to maximum allowable size’ and click ‘OK’
  3. Select the new LVM partition and click ‘Create’
  4. Select ‘LVM Volume Group’ and click ‘Create’
  5. Set the ‘Volume Group Name’ as ‘Local’ then click the ‘Add’ button
  6. Set the ‘File System Type’ as swap, the ‘Logical Volume Name’ as ‘swap’ and the ‘Size(MB)’ as ‘12288’, then click ‘OK’.
  7. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘root’, and the ‘Size(MB)’ to ‘51200’. Then click ‘OK’.
  8. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/home’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘home’, and the ‘Size(MB)’ to ‘500’. Then click ‘OK’.
  9. Click the ‘Add’ button again. Set the ‘Mount Point’ to ‘/var’, the ‘File System Type’ to ext4, the ‘Logical Volume Name’ to ‘var’, and the ‘Size(MB)’ to the remaining space available. Then click ‘OK’.
  10. Click ‘OK’
9. Click ‘Next’ and ‘Write changes to disk’ to finish the partition creation.
Leave the boot loader settings as is, and click ‘Next’
Select the ‘Minimal’ option and click ‘Next’

One of the most important things to have with servers is some form of remote management. That way you don’t need to trek down to the data center each time the server hangs while testing (and it happens a lot). For Sun systems, that means setting up the ELOM (Embedded Lights Out Manager).

Steps to set up the Remote Console (Embedded Lights Out Manager – ELOM) for SunFire X2100 M2

Set the SP serial port rate to 115200.

Log into the web based console, or through ssh via a computer on the same subnet (https://192.168.1.10) The IP is whatever the IP is set for the ELOM device. Check in BIOS for the IP.
- Go to the Configuration tab, then the Serial Port tab.
- Change the Baud Rate to 115200.

Set BIOS

IPMI Config
   Set LAN Config
   Set PEF Config
     PEF Support ........ [Enabled]
     PEF Action Global
        All of them ..... [Enabled]
     Alert Startup Discover ..... [Disabled]
     Startup Delay .............. [Disabled]
     Event Message For PEF ...... [Disabled]
   BMC Watch Dog Timer Action ... [Disabled]
   External Com Port ............ [BMC]
Remote Access
   Remote Access ................ [Serial]
   Serial Port Number ........... [Com2]
   Serial Port Mode ............. [115200 8,n,1]
   Flow Control ................. [Hardware]
   Post-Boot Support ............ [Always]
   Terminal Type ................ [VT100]
   VT-UTF8 Combo Key ............ [Enabled]

Other options for the Serial Port Mode are 9600, 19200, 38400, and 57600

Edit Linux Config Files

Add a /etc/init/serial-ttyS1.conf file

RedHat in EL 6, and thereby CentOS, moved to Upstart instead of Sysv, so we create a new serial-ttyS1.conf file instead of editing the /etc/inittab file.

#  This service maintains a getty on /dev/ttyS1.
stop on runlevel [016]

respawn
instance $TTY
exec /sbin/mingetty $TTY

Change grub.conf

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/Logical/root
#          initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
#splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

title CentOS Linux (2.6.32-71.29.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/Local-root \
rd_LVM_LV=Local/root rd_LVM_LV=Local/swap rd_NO_LUKS rd_NO_MD rd_NO_DM \
console=tty1 console=ttyS1,115200n8
          initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img

Add line to securetty

console
vc/1
vc/2
vc/3
vc/4
vc/5
vc/6
vc/7
vc/8
vc/9
vc/10
vc/11
tty1
tty2
tty3
tty4
tty5
tty6
tty7
tty8
tty9
tty10
tty11
ttyS1

SUN SP Commands

Connect to the ELOM by ssh into the IP address.
ssh root@192.168.xxx.xxx

To power on the host, enter the following command:
- set /SP/SystemInfo/CtrlInfo PowerCtrl=on

To power off the host gracefully, enter the following command:
- set /SP/SystemInfo/CtrlInfo PowerCtrl=gracefuloff

To power off the host forcefully, enter the following command:
- set /SP/SystemInfo/CtrlInfo PowerCtrl=forceoff

To reset the host, enter the following command:
- set /SP/SystemInfo/CtrlInfo PowerCtrl=reset

To reboot and enter the BIOS automatically, enter the following command:
- set /SP/SystemInfo/CtrlInfo BootCtrl=BIOSSetup

To change the IP address for the ELOM, enter:
- set /SP/AgentInfo IpAddress=xxx.xxx.xxx.xxx

The default user name is root, and the default password is changeme.
- set /SP/User/[username] Password=[password]

To start a session on the server console, enter this command:
- start /SP/AgentInfo/console
- To revert to CLI once the console has been started, press Esc-Shift-9 keys.

 To terminate a server console session started by another user, enter this command:
- stop /SP/AgentInfo/console

Next we secure the new servers with some software updates and a firewall.

Software Updates and installs:

Edit /etc/resolve.conf
nameserver 192.168.1.100
options single-request-reopen
- The last line options single-request-reopen takes care of slow SSH logins. See here https://stomp.colorado.edu/blog/blog/2011/06/29/on-rhel-6-ssh-dns-firewalls-and-slow-logins/ and here http://www.linuxquestions.org/questions/showthread.php?p=4399340#post4399340 for more info.
yum install openssh-clients tcsh ksh bc rpm-build gcc gcc-c++ redhat-rpm-config acl gcc gnupg make vim-enhanced man wget which mlocate bzip2-devel libxml2-devel screen sudo parted gd-devel pam_passwdqc.x86_64 rsync zip xorg-x11-server-utils gettext
disable SELinux. Edit the /etc/sysconfig/selinux file and set SELINUX=disabled.
- Change takes affect on next reboot.
Add the following lines to the /etc/vimrcfile:
set autoindent ” auto indent after {
set smartindent ” same
set shiftwidth=4 ” number of space characters inserted for indentation
set expandtab ” inserts spaces instead of tabs
set tabstop=4 ” number of spaces the tab is.
set pastetoggle=<C-P> ” Ctrl-P toggles paste mode
Switch root shell to tcsh
- Edit the /etc/passwdfile to have root use tcshroot:x:0:0:root:/root:/bin/tcsh
- Edit the .tcshrcfile in root’s home.
  # .tcshrc# User specific aliases and functionsalias rm ‘rm -i’
  alias cp ‘cp -i’
  alias mv ‘mv -i’set prompt='[%n@%m %c]# ‘
  
  setenv PATH ${PATH}:/opt/sun/cam/bin
  
  # Make command completion (TAB key) cycle through all possible choices
  # (The default is to simply display a list of all choices when more than one
  # match is available.)
  bindkey “^I” complete-word-fwd
- Logout and back in for it to take affect.
Edit /etc/hosts. Add a line with IP and domain name.
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6# External IPs
192.168.1.1 bill.com
192.168.1.2 ted.com192.168.1.3 domain.com # this needs to be an IP that the cluster server can manage# Internal IPs
192.168.1.11 bill.localdomain bill # notice the .localdomain, this is necessary for mysql later
192.168.1.12 ted.localdomain ted othernode # this is bill’s hosts file. othernode would be on the bill line for ted’s hosts file.
# ServicePort IPs
192.168.1.21 billsp # I like to have a short name to use to connect to the service port (ELOM)
192.168.1.22 tedsp

# Internal Services
192.168.1.100 http.localdomain httpd.localdomain
192.168.1.101 mysql.localdomain
192.168.1.102 memcached.localdomain
Run updatedb to set up the locate database.
Edit password settings to allow for stricter control over passwords. This requires strong passwords or the use of passphrases.
- Check here for updated info: http://www.nsa.gov/ia/guidance/security_configuration_guides/operating_systems.shtml#linux2
- Edit the /etc/pam.d/system-authfile. Change the linepassword requisite pam_cracklib.so try_first_pass retry=3to thispassword requisite pam_passwdqc.so min=disabled,disabled,16,12,8
[Optional] Firefox: yum update, and then ayum install firefox xorg-x11-xauth xorg-x11-fonts-Type1There will be more you’ll need too.
- If you get this error: process 702: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/var/lib/dbus/machine-id": No such file or directory. Then run the following command as root.
  - dbus-uuidgen > /var/lib/dbus/machine-id
Set up ssh keys
- ssh-keygen
- Copy the id_rsa.pub file to the other node
- Copy the contents of id_rsa.pub to cat id_rsa.pub >> ~/.ssh/authorized_keys
- Double check permission on authorized_keys and id_rsa both set to rw-------
- You should now be able to log in from bill to ted (and vice versa) without a password.

Shorewall

Yum Install:
- Get EPEL repository. Visit http://fedoraproject.org/wiki/EPEL to get the URL for the correct rpm. Something like: epel-release-6-5.noarch.rpm.
- Copy that URL and runrpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpmon the machine.
- Edit the /etc/yum.repos.d/epel.repo file and set the first “enabled” line to equal 0. That disables yum from using the EPEL repo by default.
- Install shorewall with yum.yum --enablerepo=epel install shorewall
Enable program to run by editing the /etc/shorewall/shorewall.conf file. Change the STARTUP_ENABLED=NOtoSTARTUP_ENABLED=Yes
Edit the shorewall config files.
Edit the /etc/shorewall/zonesfile:
- #
  # Shorewall version 4 – Zones File
  #
  # For information about this file, type “man shorewall-zones”
  #
  # The manpage is also online at
  # http://www.shorewall.net/manpages/shorewall-zones.html
  #
  ###############################################################################
  #ZONE TYPE OPTIONS IN OUT
  # OPTIONS OPTIONSnet ipv4 # The big bad Internet
  loc ipv4 # Internal LAN
  fw firewall#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE#LAST LINE – ADD YOUR ENTRIES ABOVE THIS ONE – DO NOT REMOVE
Edit the /etc/shorewall/interfacesfile:
- #
  # Shorewall version 4 – Interfaces File
  #
  # For information about entries in this file, type “man shorewall-interfaces”
  #
  # The manpage is also online at
  # http://www.shorewall.net/manpages/shorewall-interfaces.html
  #
  ###############################################################################
  #ZONE INTERFACE BROADCAST OPTIONS
  net eth2
  loc eth1
Edit the /etc/shorewall/policyfile:
- ###############################################################################
  #SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
  # LEVEL BURST MASK
  # To/from internal lan
  fw loc ACCEPT
  loc fw ACCEPT
  # To/from net
  fw net ACCEPT
  net all DROP info
  #
  # THE FOLLOWING POLICY MUST BE LAST
  #
  all all REJECT info
  #LAST LINE — DO NOT REMOVE
Edit the /etc/shorewall/rulesfile:
- ######################################################################################
  #ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL
  # PORT PORT DEST
  #SECTION ESTABLISHED
  #SECTION RELATED
  SECTION NEWSECTION NEW# Standard services
  #
  ACCEPT net fw tcp ssh
  ACCEPT net fw tcp 80,443Ping/ACCEPT net fw
  
  #LAST LINE — ADD YOUR ENTRIES BEFORE THIS ONE — DO NOT REMOVE
Edit the /etc/shorewall/routestoppedfile:
- #
  # Shorewall version 4 – Routestopped File
  #
  # For information about entries in this file, type “man shorewall-routestopped”
  #
  # The manpage is also online at
  # http://www.shorewall.net/manpages/shorewall-routestopped.html
  #
  # See http://shorewall.net/starting_and_stopping_shorewall.htm for additional
  # information.
  #
  ###############################################################################
  #INTERFACE HOST OPTIONS PROTO DEST SOURCE
  # PORT PORT
  eth1 –
  eth2 –
Set shorewall to start on reboots.chkconfig shorewall on
Start shorewall:service shorewall start

The next part will be connecting the servers to the storage array.

Share and Enjoy: