There are four basic steps to creating a personal website.

1. Content

You may want to start out with a cool design, or fun idea on how to interact with visitors, or what have you. But really, the most important thing a website has going for it is the content. Design is a close second (but we’ll talk about that last), because people tend to shy away from “ugly” sites. But they won’t even go in the first place if the content isn’t relevant.

You’ll need to ask yourself a few questions to get an idea of what kind of website you need. The answers will even help define the design, and determine the platform, or website technology, that you use.

• What information do you want to share?
• Why do you want to it?
• Do you want conversations to take place on your website?
• Do you want a blog, a simple website with information about you or a topic, or something else?

2. Domain Name

A domain name is a word, set of words, or phrase thatThe domain name plays an important role in search engine rankings. If this is your personal site, try to get your name, or part of it, as the domain name. It can be all you need for “brand” identification.

Shop around before you buy a domain name. There are plenty of options out there, just do a search for domain registrar. Often a hosting provider will sell domain names as well. As of this writing, you should be able to get a domain name for around $10-$11 a year. Make sure the registrar includes DNS management.

.org, .net, .com, .info, .us, … What type of domain name should you buy? There are 19 top level domains (TLDs the very last part of a domain name, the part followed by the last period), and over 250 country code top-level domains (.us, .me, .de, .uk, etc.) That depends on a few things, the most important being, which one is available. Generally, .com and .org are the most sought after. Here is a list of the top-level domains and their intended purposes (from Wikipedia).

Country code top-level domains can be used as well, often to create clever domain names (called domain hacks) like del.icio.us, bit.ly, instagr.am, pep.si, and redd.it.

source, creative commons search on flickr.com

3. Hosting

A hosting provider is the company that owns the servers where your website lives. There are many free options. Look for a hosting provider that offers “easy” installations of common software like WordPress, Drupal, etc.

• wordpress.com
• omeka.net
• weebly.com
• list of 40 free blog hosts: mashable.com/2007/08/06/free-blog-hosts
• Wikipedia: Comparison of free web hosting services

Paid options:

You can find a hosting provider from anywhere between $5/month to $100/month.

• hostgator.com
• dreamhost.com
• mediatemple.net

source: creative commons search on flickr.com

4. Design

Independent on the platform you choose, there are usually thousands of free themes available for easy download and install. When you pick a platform, look on their site for places to find free themes.

Designing a website takes lots of work to make it look nice. The better the design, the more resources are needed (be they time, or money).

Previous posts in this series:

Part 1: Setting up the servers

Part 2: Connecting the Array

RedHat Cluster and GFS2 Setup

• Required reading: https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial Seriously. Don’t do anything until you read his tutorial a couple of times. Might as well read a bunch of his other tutorials, too.
• Unless mentioned, the following commands should be done on each server.

Set date/time to be accurate and within a few minutes of each other.

• Install the ntp program and update to current time.
  • yum install ntp
  • ntpdate time.nist.gov
• Set time servers and start ntpd
  • service ntpd start
  • Edit the /etc/ntp.conf file to use the following servers:

  • server 0.pool.ntp.org
    server 1.pool.ntp.org
    server 2.pool.ntp.org
    server 3.pool.ntp.org

• Restart ntpd
  • service ntpd restart
  • chkconfig ntpd on

Cluster setup

RedHat Cluster must be set up before the GFS2 File systems can be created and mounted.

• Instal the necessary programs
  • yum install openais cman rgmanager lvm2-cluster gfs2-utils ccs
  • Create a /etc/cluster/cluster.conf REMEMBER: Always increment the “config_version” parameter in the cluster tag!

    • <?xml version=“1.0”?>
          <cluster config_version=“24” name=“web-production”>
              <cman expected_votes=“1” two_node=“1”/>
              <fence_daemon clean_start=“1” post_fail_delay=“6” post_join_delay=“3”/>
              <totem rrp_mode=“none” secauth=“off”/>
              <clusternodes>
                  <clusternode name=“bill” nodeid="1">
                      <fence>
                          <method name="ipmi">
                              <device action=“reboot” name=“ipmi_bill”/>
                          </method>
                      </fence>
                  </clusternode>
                  <clusternode name=“ted” nodeid="2">
                      <fence>
                          <method name="ipmi">
                              <device action=“reboot” name=“ipmi_ted”/>
                          </method>
                      </fence>
                  </clusternode>
              </clusternodes>
              <fencedevices>
                  <fencedevice agent=“fence_ipmilan” ipaddr=“billsp” login=“root” name=“ipmi_bill” passwd=“PASSWORD-HERE”/>
                  <fencedevice agent=“fence_ipmilan” ipaddr=“tedsp” login=“root” name=“ipmi_ted” passwd=“PASSWORD-HERE”/>
              </fencedevices>
              <rm log_level="5">
                  <resources>
                      <clusterfs device=“/dev/mapper/StorageTek2530-sites” fstype=“gfs2” mountpoint=“/sites” name=“sites”/>
                      <clusterfs device=“/dev/mapper/StorageTek2530-databases” fstype=“gfs2” mountpoint=“/databases” name=“databases”/>
                      <clusterfs device=“/dev/mapper/StorageTek2530-logs” fstype=“gfs2” mountpoint=“/logs” name=“logs”/>
                  </resources>
                  <failoverdomains>
                      <failoverdomain name=“bill-only” nofailback=“1” ordered=“0” restricted="1">
                          <failoverdomainnode name=“bill”/>
                      </failoverdomain>
                      <failoverdomain name=“ted-only” nofailback=“1” ordered=“0” restricted="1">
                          <failoverdomainnode name=“ted”/>
                      </failoverdomain>
                  </failoverdomains>
              </rm>
          </cluster>

  • We’ll be adding more to this later, but this will work for now.
  • Validate the config file
    • ccs_config_validate
  • Set a password for the ricci user
    • passwd ricci
  • Start ricci, and set to start on boot
    • service ricci start
    • chkconfig ricci on
  • Start modclusterd and set to start on boot
    • service modclusterd start
    • chkconfig modclusterd on
  • Sync the cluster.conf file to other node
    • ccs -f /etc/cluster/cluster.conf -h ted --setconf
  • Start cman on both servers at the same time
    • service cman start
  • Set cman to start on boot
    • chkconfig cman on
• Check the tutorial on testing the fencing
  • https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Testing_Fencing

Create GFS2 partitions

Create a partition on the new scsi device /dev/mapper/mpatha using parted. NOTE: This part only needs to be done once on one server

• parted /dev/mapper/mpatha
• mklabel gpt
• mkpart primary 1 -1
• set 1 lvm on
• quit
• Now you can see a partition for the storage array.
  • parted -l

Edit the /etc/lvm/lvm.conf file and set the value for locking_type = 3 to allow for cluster locking.

In order to enable the LVM volumes you are creating in a cluster, the cluster infrastructure must be running and the cluster must be quorate.

• service clvmd start
• chkconfig clvmd on
• chkconfig gfs2 on

Create LVM partitions on the raw drive available from the StorageTek. NOTE: This part only needs to be done once on one server.

• pvcreate /dev/mapper/mpatha1
• vgcreate -c y StorageTek2530 /dev/mpatha1

Now create the different partitions for the system: sites, databases, logs, home, root

• lvcreate --name sites --size 350GB StorageTek2530
• lvcreate --name databases --size 100GB StorageTek2530
• lvcreate --name logs --size 50GB StorageTek2530
• lvcreate --name root --size 50GB StorageTek2530

Make a temporary directory /root-b and copy everything from root’s home directory to there, because it will be erased when we make the GFS2 file system.

Copy /root/.ssh/known_hosts to /etc/ssh/root_known_hosts so the file is different for both servers.

Before doing the home directory, we have to remove it from the local LVM.

• unmount /home
• lvremove bill_local/home and on ted lvremove ted_local/home
• Remove the line from /etc/fstab referring to the /home directory on the local LVM
• Then add the clustered LV.
  • lvcreate --name home --size 50GB StorageTek2530

Create GFS2 files systems on the LVM partitions created on the StorageTek. Make sure they are unmounted, first. NOTE: This part only needs to be done once on one server.

• mkfs.gfs2 -p lock_dlm -j 2 -t web-production:sites /dev/mapper/StorageTek2530-sites
• mkfs.gfs2 -p lock_dlm -j 2 -t web-production:databases /dev/mapper/StorageTek2530-databases
• mkfs.gfs2 -p lock_dlm -j 2 -t web-production:logs /dev/mapper/StorageTek2530-logs
• mkfs.gfs2 -p lock_dlm -j 2 -t web-production:root /dev/mapper/StorageTek2530-root
• mkfs.gfs2 -p lock_dlm -j 2 -t web-production:home /dev/mapper/StorageTek2530-home

Mount the GFS2 partitions

• NOTE: GFS2 file systems that have been mounted manually rather than automatically through an entry in the fstab file will not be known to the system when file systems are unmounted at system shutdown. As a result, the GFS2 script will not unmount the GFS2 file system. After the GFS2 shutdown script is run, the standard shutdown process kills off all remaining user processes, including the cluster infrastructure, and tries to unmount the file system. This unmount will fail without the cluster infrastructure and the system will hang.
• To prevent the system from hanging when the GFS2 file systems are unmounted, you should do one of the following:
  • Always use an entry in the fstab file to mount the GFS2 file system.
  • If a GFS2 file system has been mounted manually with the mount command, be sure to unmount the file system manually with the umount command before rebooting or shutting down the system.
• If your file system hangs while it is being unmounted during system shutdown under these circumstances, perform a hardware reboot. It is unlikely that any data will be lost since the file system is synced earlier in the shutdown process.

Make the appropriate folders on each node (/home is already there).

• mkdir /sites /logs /databases

Make sure the appropriate lines are in /etc/fstab

#GFS2 partitions shared in the cluster
/dev/mapper/StorageTek2530-root        /root        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-home        /home        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-databases      /databases      gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-logs        /logs        gfs2   defaults,acl    0 0
/dev/mapper/StorageTek2530-sites    /sites    gfs2   defaults,acl    0 0

Once the GFS2 partitions are set up and in /etc/fstab, rgmanager can be started. This will mount the GFS2 partions.

• service rgmanager start
• chkconfig rgmanager on

Starting Cluster Software

To start the cluster software on a node, type the following commands in this order:

• service cman start
• service clvmd start
• service gfs2 start
• service rgmanager start

Stopping Cluster Software

To stop the cluster software on a node, type the following commands in this order:

• service ossec-hids stop
  • (ossec monitors the apache log files, so the /logs partition will not be unmounted unless ossec is stopped first.)
• service rgmanager stop
• service gfs2 stop
• umount -at gfs2
• service clvmd stop
• service cman stop

Cluster tips

If a service shows as ‘failed’ when checking on services with clustat

• Disable the service first: clusvcadm -d service-name
• Then re-enable it: clusvcadm -e service-name

Have Shorewall start sooner in the boot process.

• It was necessary to move shorewall up in the boot process, otherwise cman had no open connection to detect the other nodes.
• Edit the /etc/init.d/shorewall and change the line near the top from # chkconfig: - 28 90 to
  • # chkconfig: - 18 90
• Then use chkconfig to turn off shorewall and then back on.
  • chkconfig shorewall off
  • chkconfig shorewall on

Not quite the same feeling and fun as a road trip, but fun enough.
Yeah, archive work! Yeah, Germany! Yeah, yeah archive work in Germany! So I just spent the last two weeks in Germany (by myself, not so yeah) doing some archival research for the dissertation. Here are some thoughts on the trip.

1. Internet!

The Starbucks with Internet… saved my bacon.

Make sure you have a good internet connection where you will stay. I booked a decent hotel with Internet included, and free breakfast. The only problem is, the connection to the Internet is spotty at best. I have to get a new user/pass combination to connect to the Internet every 24 hours, too. It’s so frustrating to want to communicate, but not be able to. Especially when you’re trying to get in touch with family back home. So, do some research and hope you get lucky. Also, it is important to know any quirks about the Internet in the country you go to, if going out side of the USA. In Germany, they use 13 channels for their routers, in the USA we only use 11. So if your place of stay uses channel 12 or 13, you’re almost out of luck. You can pick up a relatively cheap USB wireless adapter in the country that should get you all of their available channels. But you will most likely have to find some place with Internet to download software. Enter in the great Internet hubs scattered throughout the world: Starbucks and McDonalds! Even BurgerKing has Internet available. Find out where they are and use them.

2. This is only a test.

Getting ready for Christmas!

Don’t get your hopes up too high for your first trip. I kind of went on this trip with the attitude that it would be a test run of a later real trip. This was possible because I know that I’m coming back in a few months. If you don’t know if you’ll ever go back, then you need to do a lot of background research and contacting before hand. I had scheduled to go to the archive Tuesday through Friday the first week and Monday through Thursday the second week. The first day ended up being a get settled day; exchanging money, finding my way around, finding the Starbucks for Internet, etc. It often felt like I was wasting time, but if you know you are going to go back, then it is time well spent to get your bearings and figure things out. I lived in Germany for two years, but that was a life time ago (like 15 years ago). So I am a bit rusty on speaking German, and German customs, and such. Luckily that mostly all came back easily.

3. Talk to me.

A house divided… will make a good restaurant.

Talk with your contacts before leaving home. Or email them. Let them know exactly what you want to do, what you want to research, where you are going to look, etc. They can save you lots of time. I had one contact at the University of Freiburg, Professor Ulricht Herbert. I met with him twice, and he gave me sound advice. I should have emailed him more often before hand, but nothing really beats face to face contact anyway. My one contact here has turned into two or three. He also helped me realize I am trying to do too much in my dissertation. As it stands, its really a life’s work project. Going through the sources helped me understand that too. There is just way too much for me to be able to grasp it in two years time (my goal). Instead, I’m going to scale back and only cover one tunnel project, and cover that in depth. The reason there is no all encompassing history about the underground projects from World War II is because it was a huge project. Basically the whole of the German economy was turned to focus on these projects towards the end of the War. There is just too much to understand, too many documents to go through, and too much to grasp before this history can be written. That’s why nobody has done it, yet. It would take lots of financing and lots of time. Dr. Herbert suggested four years of work, but only after I had perfect understanding of German, have read all that has been written on the subject so far, and had an intimate grasp of Germany in World War II. That ain’t gonna happen in two years when I have a full-time job, a family, and no financial support. Perhaps that will be my ongoing project as a professor…

4. I’ll make a note of that…

Hotel Schemmer. Home away from home.

Figure out a good note taking routine. I have several hundred documents digitized from another archive already, and figured out a good naming scheme for them. I have a spread sheet for taking notes on each file, and for later import into Omeka for an online archive. This time around was a little different. The Bundesarchiv-Militärarchiv in Freiburg had lots of documents for me to go through. Whereas before it was one collection/folder in one archive, I now had many collections/folders in one archive. So I had to figure out something a little different. I also didn’t have enough money to make digital copies of any of the records I found. It turned out that I didn’t need to make any, but that should be budgeted and planned for as well. There were a handful of documents that I wanted copies of, so I just transcribed them into a word processing document. I thought about making them plain text documents, but ran into a few formatting issues. I chose to make them LibreOffice (OpenOffice) Text documents, because there will always be a program that can open those, and that program is free. Of course, any program nowadays can open Microsoft Word documents, too, and there is no fancy formatting, so that would work fine too. One of my greatest struggles so far is keeping the documents in place chronologically. So my naming scheme for the files takes care of that. Start the name of the file with the year, then month number, then day number (YYYY-MM-DD), and the documents sort themselves! The file viewer (File Explorer for Windows or Finder for Macs) will usually sort by alphabet, so there’s nothing to it. Another thing I did was to go through the documents as quickly as I could. If It looked like it was helpful, I jotted notes about it, or quickly transcribed it. I will be able to go through the notes and transcriptions later to make sense out of them. That leads into the next point.

5. Plan it right.

Trolly going through the tunnel.

Leave a day on either end for miscellaneous things. I unintentionally had a whole day with nothing to do. I was finished with the archives on Wednesday, and didn’t need to leave until Friday. That left me with the whole day on Thursday to tie things up and get ready to leave. I did some laundry, packed my bags and wrote this. It’s also a good time to go through the notes to make sure you don’t forget anything.

6. Enjoy!

The final tip is to just enjoy the time. If you’re in a foreign country, take a day to go see the sights. I had a weekend where the archive was not even open, so I spent the day walking around the awesome Altstadt (the oldest part of town, buildings from the 1400′s!). If you have funding for your trip, just think, who else gets paid to go look at old documents. Man, history is great!

…copy somebody’s code. Or at least your own from the past that works.

I finally got the map to work for my Exhibit exhibit. See.

First I copied the tutorial here. When that was working, I carefully copied it to the file I wanted it to be in and… nothing. Still broken.

So next I commented out all extraneous stuff, and finally got it to work. Now came the fun part of tracking down what code was making my awesome map look like nothing. I narrowed it down to something within the <header> tag. So line by line I uncommented and tested until I tracked the issue down to the bootstrap.css file I was using.

Then came the same process within the bootstrap.css file. It was easily 1000 lines long, so a line by line process wasn’t going to work. Instead I commented out the first have of the file. Bing, the map displayed, so it was somewhere within that first 500 lines. Then I commented out the first 250 lines. Bing again. I kept dividing the culprits in half until I narrowed it down to styles for the img tag. Commented those out and I was in business.

Through that grueling process I lost the images of the documents on the first page. Now I have to figure out how to get those back, apply some styling and we’re all set.

Unfortunately I wasn’t ever able to get it to pull in the data from a spreadsheet, so the JSON file will have to do. The process for making a JSON file from a spreadsheet is simple:

1. Make the spreadsheet file so that the headers are encapsulated with curly braces ‘{ }’

2. If you’re using Google spreadsheets, export it as an Excel file. You’ll have to open it in Excel and save it as an older file format, because the next step doesn’t like .xlsx.

3. Next, use Similie’s Babel tool to convert the spreadsheet to JSON.

4. Select “Excel” as the “from format” and “Exhibit JSON” as the “to format” and then browse to find the xls file you just made.

5. Click the “Upload and Convert” button.

6. The next screen is all of your spreadsheet data output into a nice JSON format.

7. Now you can use this JSON file for any number of things. But it’s especially nice to use it in an Exhibit to display your data.

This is cross posted at the class blog http://www.fredgibbs.net/clio3workspace/blog/if-at-first-you-dont-succeed/

As I’m sure many people have run into this before, and from personal experience, found nothing roaming the Interwebs on how to fix it, and seeings how I just fixed it, I’ll write up a little how to fix it post.

The set up:

Two SunFire X2100 M2 servers connected to a StorageTek 2530 via iscsi. The two nodes are running CentOS 6.2 with RedHat cluster software. I have a server running Nagios for monitoring and it checks for failed disks on the StorageTek by running a script on either node that returns the number of “optimal” and “failed” disks.

Two nodes connected to an array via iscsi (Ethernet)

The problem:

Updating system software is important. Keeping packages up to date protects from security vulnerabilities. Unfortunately, sometimes it breaks things. In this case, updating the suggested packages broke my set up, making it so that the Sun Storage CAM (Common Array Manager) software did not work anymore.

I first became alerted to this when Nagios sent me errors from the script checking the StorageTek disks. I checked the command that runs in the script to see what was up, and it returned several errors. Here they are for future googlers:

sscs list -i 192.168.128.101 device

returned “Command failed due to an exception. null” and

sscs list -a arrayname host

returned “arrayname : The resource was not found.”

Not particularly helpful messages.

Fortunately the nodes could still mount the array partitions, which allowed them to continue running as web and mysql servers. I just couldn’t run management commands on the array.

Since some of the required software for CAM was updated, I supposed that was causing the issue. The required software is listed below:

• libXtst-1.0.99.2-3.el6.i686.rpm and its dependent rpm (InstallShield requirement)
• libselinux-2.0.94-2.el6.i686.rpm
• audit-libs-2.0.4-1.el6.i686.rpm
• cracklib-2.8.16-2.el6.i686.rpm
• db4-4.7.25-16.el6.i686.rpm
• pam-1.1.1-4.el6.i686.rpm
• libstdc++-4.4.4-13.el6.i686.rpm
• zlib-1.2.3-25.el6.i686.rpm
• ksh-20100621-2.el6.x86_64.rpm

The solution:

I couldn’t figure out on my own exactly what was wrong, so I contacted Oracle support, and they finally tipped me off to the solution. Completely remove the CAM software and reinstall it. Those steps are outlined below:

• Go to the CAM software folder in /var/opt/CommonArrayManager/Host_Software_6.9.0.16/bin/and run

./uninstall -f

• Here is a good spot to run yum update and restart the server if needed.
• Change directories to where you have the CAM software install CD. There should be a folder called components in there. Change into that directory and install the jdk available there:

rpm -Uvh jdk-6u20-linux-i586.rpm

• Next run the RunMe.bin file in the CAM Software CD folder.

./RunMe.bin -c

• Install the RAID Proxy Agent package located in the Add_On/RaidArrayProxy directory of the latest CAM software distribution.

rpm -ivh SMruntime.xx.xx.xx.xx-xxxx.rpm

rpm -ivh SMagent-LINUX-xx.xx.xx.xx-xxxx.rpm

• Register the array with the host/node. This process can take several minutes.

sscs register -d storage-system

One additional issue I ran into, was that some update or other process shutdown the NIC connecting the node to the array. I had to make sure that was running before I ran the register -d storage-system command above.

I’m trying this new idea for backing up our production MySQL servers. I have a backup server that basically runs rdiff-backup in the morning across several servers, but then does nothing for the rest of the day. It’s a pretty decent machine, so I’d like to utilize some resources. Replicating a MySQL server is a good way to ensure High Availability in case of a failure. The backup server acts as a slave to the master (production) server. Basically, the slave is an exact copy of the master. They are two separate instances of MySQL server running on two physical servers. Whatever queries run on the master are sent to the slave so it can do the same. This way they are kept completely in sync. You could also have the slave take over for the master, should the master server happen to fail.

The slave is an ever updating duplicate of the master.

The only problem I face with this set up, though, is that I have multiple production servers out there. So this only works if this backup server could be a slave for multiple machines.

No slave can serve two masters.

This is not possible, though, because, of course, no slave can serve two masters. Fortunately, a server can have multiple instances of MySQL running on it! So, in a sense, we have a server with multiple MySQL instances, to which a master can replicate. More about that set up in an upcoming post.

The slave has multiple instances of MySQL running.

A how to on this blog, shows how this can be done. I’ll replicate the process below.

STEPS TO MULTIPLE MYSQL MADNESS

On the slave server

Step 1. Install MySQL

We’ll be working with CentOS 5.8, but this could really apply for any OS. First we’ll need to install MySQL like normal.

yum install mysql mysql-server

There are plenty of good tutorials out there on how to install the specific version of MySQL you want on the specific OS you’re running.

Step 2. Set up the data area.

You’ll need to have a different folder for each of the MySQL instances, say /dbases/master-a/, /dbases/master-b/, and /dbases/master-c/.

mkdir -p /dbases/{master-a,master-b,master-c}

Step 3. Copy the default my.cnf file

This is the default MySQL config file, it may be named differently on other OSes.

cp /etc/my.cnf /etc/master-a.cnf; cp /etc/my.cnf /etc/master-b.cnf; cp /etc/my.cnf /etc/master-c.cnf

Step 4. Edit the new MySQL config files.

For each new config file, you’ll need to specify some unique variables.

[mysqld]
port=3307
datadir=/dbases/master-a
socket=/dbases/master-a/mysql.sock
user=mysql
server_id=3307
log-bin=/dbases/master/mysql-bin.log

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links=0

[mysqld_safe]
log-error=/dbases/master-a/mysqld.log
pid-file=/dbases/master-a/mysqld.pid

The port option sets this MySQL instance on a different port than the default 3306. The datadir, socket, log-bin, log-error, and pid-file options make sure the necessary files are not using the default files.

Step 5. Create new init scripts.

The init script allows the server to start and stop the service at boot time, and allows for easy start up and shutdown (on CentOS/RedHat, at least – with an easy service mysqld start).

cp /etc/init.d/mysqld /etc/init.d/mysqld-master-a

Just do one for now. We’ll copy the new one to create the others, then just do a quick search and replace in those files to change the master-a to master-b and master-c.

Step 6. Edit the init script

#!/bin/bash
#
# mysqld        This shell script takes care of starting and stopping
#               the MySQL subsystem (mysqld).
#
# chkconfig: - 64 36
# description:  MySQL database server.
# processname: mysqld
# config: /etc/master-a.cnf
# pidfile: /dbases/master-a/mysqld.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

prog="MySQL"

# extract value of a MySQL option from config files
# Usage: get_mysql_option SECTION VARNAME DEFAULT
# result is returned in $result
# We use my_print_defaults which prints all options from multiple files,
# with the more specific ones later; hence take the last match.
get_mysql_option(){
        result=`/usr/bin/my_print_defaults "$1" | sed -n "s/^--$2=//p" | tail -n 1`
        if [ -z "$result" ]; then
            # not found, use default
            result="$3"
        fi
}

get_mysql_option mysqld datadir "/dbases/master-a"
datadir="/dbases/master-a"
get_mysql_option mysqld socket "/dbases/master-a/mysql.sock"
socketfile="/dbases/master-a/mysql.sock"
get_mysql_option mysqld_safe log-error "/dbases/master-a/mysqld.log"
errlogfile="/dbases/master-a/mysqld.log"
get_mysql_option mysqld_safe pid-file "/dbases/master-a/mysqld.pid"
mypidfile="/dbases/master-a/mysqld.pid"

defaultfile="/etc/master-a.cnf"

start(){
        touch "$errlogfile"
        chown mysql:mysql "$errlogfile"
        chmod 0640 "$errlogfile"
        [ -x /sbin/restorecon ] && /sbin/restorecon "$errlogfile"
        if [ ! -d "$datadir/mysql" ] ; then
            action $"Initializing MySQL database: " /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql
            ret=$?
            chown -R mysql:mysql "$datadir"
            if [ $ret -ne 0 ] ; then
                return $ret
            fi
        fi
        chown mysql:mysql "$datadir"
        chmod 0755 "$datadir"
        # Pass all the options determined above, to ensure consistent behavior.
        # In many cases mysqld_safe would arrive at the same conclusions anyway
        # but we need to be sure.
        /usr/bin/mysqld_safe  --defaults-file="$defaultfile" --datadir="$datadir" --socket="$socketfile" \
                --log-error="$errlogfile" --pid-file="$mypidfile" \
                --user=mysql >/dev/null 2>&1 &
        ret=$?
        # Spin for a maximum of N seconds waiting for the server to come up.
        # Rather than assuming we know a valid username, accept an "access
        # denied" response as meaning the server is functioning.        
        if [ $ret -eq 0 ]; then
            STARTTIMEOUT=30
            while [ $STARTTIMEOUT -gt 0 ]; do
                RESPONSE=`/usr/bin/mysqladmin --socket="$socketfile" --user=UNKNOWN_MYSQL_USER ping 2>&1` && break
                echo "$RESPONSE" | grep -q "Access denied for user" && break
                sleep 1
                let STARTTIMEOUT=${STARTTIMEOUT}-1
            done
            if [ $STARTTIMEOUT -eq 0 ]; then
                    echo "Timeout error occurred trying to start MySQL Daemon."
                    action $"Starting $prog: " /bin/false
                    ret=1
            else
                    action $"Starting $prog: " /bin/true
            fi
        else
            action $"Starting $prog: " /bin/false
        fi
        [ $ret -eq 0 ] && touch /dbases/master-a/mysqld
        return $ret
}

stop(){ 
        MYSQLPID=`cat "$mypidfile"  2>/dev/null `
        if [ -n "$MYSQLPID" ]; then
            /bin/kill "$MYSQLPID" >/dev/null 2>&1
            ret=$?
            if [ $ret -eq 0 ]; then
                STOPTIMEOUT=60
                while [ $STOPTIMEOUT -gt 0 ]; do
                    /bin/kill -0 "$MYSQLPID" >/dev/null 2>&1 || break
                    sleep 1
                    let STOPTIMEOUT=${STOPTIMEOUT}-1
                done
                if [ $STOPTIMEOUT -eq 0 ]; then
                    echo "Timeout error occurred trying to stop MySQL Daemon."
                    ret=1
                    action $"Stopping $prog: " /bin/false
                else
                    rm -f /dbases/master-a/mysqld
                    rm -f "$socketfile"
                    action $"Stopping $prog: " /bin/true
                fi
            else
                action $"Stopping $prog: " /bin/false
            fi
        else
            ret=1
            action $"Stopping $prog: " /bin/false
        fi
        return $ret
}

restart(){
    stop
    start
}

condrestart(){
    [ -e /dbases/master-a/mysqld ] && restart || :
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status mysqld
    ;;
  restart)
    restart
    ;;
  condrestart)
    condrestart
    ;;
  *)
    echo $"Usage: $0 {start|stop|status|condrestart|restart}"
    exit 1
esac

exit $?

Step 7. Start each MySQL instance.

Now you can start each instance with the handy service command.

service mysqld-master-a start

Step 8. Connect to MySQL instances.

Now, to connect to each MySQL instance, you’ll need to specify the port and/or socket file.

mysql -P3307 --socket="/dbases/mysql-master-a/mysql.sock"

I’m trying this new idea for backing up our production MySQL servers. I have a backup server that basically runs rdiff-backup in the morning across several servers, but then does nothing for the rest of the day. It’s a pretty decent machine, so I’d like to utilize some resources. Databases are a tough cookie to backup. You can’t just copy the data files and then expect to copy them back over and have them just work. Especially if your databases have a mixture of InnoDB and MyISAM tables. In order to do a clean and accurate database backup, you need to stop the MySQL server, then copy the files, then restart MySQL.

If you have a live production MySQL server, stopping it to make a backup is not really an option. Fortunately there are a few options. Before you decide on which option to choose, here is a list of things to keep in mind when choosing a backup solution (from the MySQL gurus at Percona):

WHAT TO LOOK FOR

http://www.mysqlperformanceblog.com/2009/03/03/10-things-you-need-to-know-about-backup-solutions-for-mysql/

1. Does the backup require shutting down MySQL? If not, what is the impact on the running server? Blocking, I/O load, cache pollution, etc?
2. What technique is used for the backup? Is it mysqldump or a custom product that does something similar? Is it a filesystem copy?
3. Does the backup system understand that you cannot back up InnoDB by simply copying its files?
4. Does the backup use FLUSH TABLES, LOCK TABLES, or FLUSH TABLES WITH READ LOCK? These all interrupt processing.
5. What other effects are there on MySQL? I’ve seen systems that do a RESET MASTER, which immediately breaks replication. Are there any FLUSH commands at all, like FLUSH LOGS?
6. How does the system guarantee that you can perform point-in-time recovery?
7. How does the system guarantee consistency with the binary log, InnoDB logs, and replication?
8. Can you use the system to set up new MySQL replication slaves? How?
9. Does the system verify that the backup is restorable, e.g. does it run InnoDB recovery before declaring success?
10. Does anyone stand behind it with support, and guarantee working, recoverable backups? How strong is the legal guarantee of this and how much insurance do they have?

BACKUP PROGRAMS

There are a few MySQL backup products out there as well. I have used the first two on this list.

• AutoMySQLBackup script (handy for making a rotating incremental backup of your MySQL databases).
• Percona XtraBackup (nice way to ensure InnoDB and MyISAM tables are backed up properly, also does it incrementally)
• Zmanda (seems to be similar to Percona’s set up)

There’s probably a gazillion more out there. Google’s your friend in finding things you need.

HOW TO DO IT

How to get a copy of the master to the slave?

There are several options. You could use a script above, or create a slave of the database (basically an exact copy of the production MySQL server – all changes that occur in the master are sent to the slave), or some combination. I’ll use a combination. I’ll replicate the production server onto the backup server, then run the incremental backups from there. This first part will walk through the process of setting up MySQL replication.

To give proper credit, here are several other how to’s I found helpful.

• http://www.badllama.com/content/mysql-backups-using-lvm-snapshots
• http://www.mysqlperformanceblog.com/2006/08/21/using-lvm-for-mysql-backup-and-replication-setup/
• http://www.softwareprojects.com/resources/programming/t-how-to-move-copy-a-live-mysql-database-and-what-1257.html
• http://www.clusterdb.com/mysql-cluster/get-mysql-replication-up-and-running-in-5-minutes/

On the master server

Step 1. Edit the my.cnf file to include at least the following (if needed) lines. Note: you will have to restart MySQL for these changes to take affect.

[mysqld]
server_id=1
innodb_flash_log_at_trx_commit=1
log_bin=mysql-bin.log
sync_binlog=1

Step 2. Make a MySQL user for the slave to use.

GRANT REPLICATION SLAVE ON *.* TO 'rep_user'@'localhost' IDENTIFIED BY 'passwordhere';

Step 3. Open a terminal session and log in to a MySQL prompt. Type the following command and hit enter.

FLUSH TABLES WITH READ LOCK

Step 4. After the FLUSH TABLES command finishes, run the following command and press enter.

SHOW MASTER STATUS

Step 5.  Make a copy of the database files.

lvcreate -L10G -s -n mysql-backup /dev/mapper/dbases

mkdir -p /mnt/mysql-backup

mount -o nouuid /dev/mapper/mysql-backup /mnt/mysql-backup

rsync -avz -e "ssh -c blowfish" /mnt/mysql-backup user@remote.host:/backup/location

rsync -avz -e "ssh -c blowfish" /dbases/mysql user@remote.host:/backup/location

mysqldump -uuser -p --all-databases > mysql-backup.sql

scp mysql-backup.sql user@remote.host:/backup/location

Step 6. Once the lvcreate command has finished, you can unlock the database.

UNLOCK TABLES

Step 7. If you haven’t already, copy the copy of the database files to the backup server.

On the slave server

Step 1. Edit the my.cnf file to include at least the following (if needed) lines. Note: you will have to restart MySQL for these changes to take affect.

[mysqld]
server_id=2

Step 2. Start MySQL and run the following commands in a mysql session to start the MySQL slave.

CHANGE MASTER TO
MASTER_HOST = "master.server.com",
MASTER_USER = "rep_user",
MASTER_PASSWORD = "passwordhere",
MASTER_LOG_FILE = "mysql-bin.log",
MASTER_LOG_POS = 2341234;

START SLAVE;

Believe it or not, this was my first ever THATCamp experience. Seems hard to imagine since I have worked at CHNM since before the inception of THATCamp.  Ah, well, the stars finally aligned, and was able to attend this year. And it was great.

I attended three workshops on Friday, and a couple of sessions on Saturday. Here are some thoughts…

Digital Data Mining with Weka: This was a neat crash course on what data mining is and is not, and one free tool to help do some of that. It was more of a “here’s what’s out there, how does it apply to the humanities” than a hands on, hands dirty, do some real work type of work shop. But it was good in that it opened up the possibility to do some data mining in the future. Since they were relatively small, here my notes:

What is data mining?

examples:

• recommend ads, products to users based on purchases
• finding patterns in large amounts of text (writing algorithms to find those patterns)

Goal of data mining is to predict what users/people will do based on data from other users/people

Data mining Tasks: classification, clustering, association rule discovery, sequential pattern discovery, regression, deviation detection

Goal is to predict the classification (or whatever) of future data, based on the current data that you analyze.

Cross validation (when done correctly you get true results, when not done correctly you get false results)

have multiple sets of data, one for training and the other for testing. Build the algorithm on the training data, then run it on the test. Then cycle through each testing data set and have it act as a training set. Do this way because you know the results for each, so you can tell if your algorithm is correct. When it’s good then you can use it on future data where you don’t know the result.

Interesting Things You Can Do With Git: This one was highly anticipated. I have been wanting/needing to learn git for a while now. For being in the IT field, having written some code, and even having a GitHub account with code on it, I’m ashamed to say I still don’t know how to use git effectively. There is not much you can do in 1.5 hours, and this was more a theoretical “here are some ideas”, than a “here is how to do it” approach.

The session on using blogs as assignments was maybe a bit premature for me. The session was really good, great ideas, tips, etc. But me teaching is still too far away for me to have put the mental effort in to following along much. I spent most of the time trying to find a good twitter client, but in the end just stuck with Adium.

Then I took some time to enjoy the hacker space. I decided it was time, and this was the perfect place to set up a transcription tool for my dissertation archive. So sitting at the very table where it is coded, I installed the Scripto plugin for Omeka. That’s a bit of a misnomer, since it is really a wrapper that lets Omeka and a Wikimedia install play nicely together. I went ahead and transcribed one of the documents in the archive as well. The archive is just in the testing phase, but here it is anyways: http://nazitunnels.org/archive

Nazi Tunnels Archive

The final event of THATCamp for me was one last session proposed by a fellow “camper”. She wanted help learning about the shell/terminal/command line. So I volunteered to help out with that. It ended up that there were about eight people that wanted help learning the command line, and four of us that knew what we were doing. So it ended up being a great ratio of help needed to those who could offer it. We started with the very basics, didn’t get much past a few commands (ls, cd, rm, nano, grep, cat), but we went slow enough that everybody who was trying to follow along was able to, and they all left with a clearer understanding of what the shell is for, and why it is useful. The proposer found a great tutorial/book for learning the command line that I’ll have to go through as well. You can always learn something new.

What was also great about that session, since it was basically ran by those who needed the help, I saw how those who struggle with these concepts learn them, so I will hopefully be better able to teach them to others in the future.

UPDATE: I forgot to mention the many cool sites and projects mentioned during Saturday morning’s Dork Shorts. Here’s a list of the ones I took notice of.

http://afripod.aodl.org/
https://github.com/AramZS/twitter-search-shortcode
http://cowriting.trincoll.edu/
http://www.digitalculture.org/2012/06/15/dcw-volume-1-issue-4-lib-report-mlaoa-and-e-lit-pedagogy/
http://luna.folger.edu/luna/servlet/BINDINGS~1~1
http://www.insidehighered.com/blogs/gradhacker
http://hacking.fugitivetexts.net/
http://jitp.commons.gc.cuny.edu/
http://www.neh.gov/
http://penn.museum/
http://www.playthepast.org/
http://anglicanhistory.org/
http://podcast.gradhacker.org/
http://dhcommons.org/
http://ulyssesseen.com/
http://m.thehenryford.org/

Doh!

Sometimes AWStats will miss some days in calculating stats for your site, and that leaves a big hole in your records. Usually, as in my case, it’s because I messed up. I reinstalled some software on our AWStats machine, and forgot to reinstall cron. Cron is the absolutely necessary tool for getting the server to run things on a timed schedule. I didn’t notice this until several days later, leading to a large gap in the stats for April.

What to do?

Fortunately, there is a fix. Unfortunately, it’s a bit labor intensive, and depends on how you rotate your apache logs (if at all, which you should). The AWStats Documentation (see FAQ-COM350 and FAQ-COM360) has some basic steps to fix the issue, outlined below:

1. Move the AWStats data files for months newer to a temporary directory.
2. Copy the Apache logs with all of the stats for the month with the missing days to a temporary directory.
3. Run the AWStats update tool, using AWStat’s logresolvemerge tool and other changed paramaters, to re-create the AWStats data file for that month
4. Replace the AWStats data files for the following months (undo step 1).

The Devil’s in the Details

Again, depending on how you have Apache logs set up, this can be an intensive process. Here’s how I have Apache set up, and the process I went through to get the missing days back into AWStats.

We have our Apache logs rotate each day for each domain on the server (or sub-directory that is calculated separately). This means I’ll have to do this process about 140 times. Looks like I need to write a script…

Step 1. Move the data files of newer months

AWStats can’t run the update on older months if there are more recent months located in the data directory. So we’ll need to move the more recent month’s stats to a temporary location out of the way. So, if the missing dates are in June, and it is currently August, you’ll need to remove the data files for June, July, and August (they look like this awstatsMMYYYY.domain-name.com.txt where MM is the two digit month and YYYY is the four digit year) to a temporary directory so they are out of the way.

Step 2. Get the Apache logs for the month.

First step is to get all of the logs for each domain for the month. This will work out to about 30 or 31 files (if the month is already past), or however many days have past in the current month. For me, each domain archives the days logs in the following format domain.name.com-access_log-X.gz and domain.name.com-error_log-X.gz where the X is a sequential number. So the first problem is how to get the correct file name without having to look in each file to see if it has the right day? Fortunately for me, nothing touches these files after they are created, so their mtime (the time stamp of when they were last modified) is intact and usable. Now, a quick one-liner to grab all of the files within a certain date range and put their content in a new file.

We’ll use the find command to find the correct files. Before we construct that command, we’ll need to create a couple of files to use for our start and end dates.

touch --date YYYY-MM-DD /tmp/start

touch --date YYYY-MM-DD /tmp/end

Now we can use those files in the actual find command. You may need to create the /tmp/apachelogs/ directory first.

find /path/to/apache/logs/archive/ -name "domain-name.com-*" -newer /tmp/start -not -newer /tmp/end -exec cp '{}' /tmp/apachelogs/ \;

Now unzip those files so they are usable. Move into the /tmp/apachelogs/ directory, and run the gunzip command.

gunzip *log*

If you are doing the current month, then copy in the current apache log for that domain.

cp /path/to/apache/logs/current/domain-name.com* /tmp/apachelogs/

This puts all of the domains log files for the month into a directory that we can use in the AWStats update command

Things to note: You need to make sure that each of the log files you have just copied use the same format. You also need to make sure they only contain data for one month. You can edit the files by hand or throw some fancy sed commands at the files to remove any extraneous data.

Step 3. Run the AWStats logresolvemerge and update tool

Now comes the fun part. We first run the logresolvemerge tool on the log files we created in the previous step to create one single log file for the whole month. While in the <code>/tmp/apachelogs/</code> directory, run:

perl /path/to/logresolvemerger.pl *log* > domain-name.com-YYYY-MM-log

Now, we need to run the AWStats update tool with a few parameters to account for the location of the new log file.

perl /path/to/awstats.pl -update -configdir="/path/to/awstats/configs" -config="domain-name.com" -LogFile="/tmp/apachelogs/domain-name.com-YYYY-MM-log"

Step 4. Move back any remaining files

If you moved any of the AWStats data files (awstatsMMYYYY.domain-name.com.txt like for July and August in our example) now’s the time to move them back where they go.

Yeah, that fixed it!

Phew! The missing dates are back!

Scrivener is awesome software for writing, that I’ve mentioned before, but I had yet to really test out the integration with Zotero (my citation manager of choice). So now that I have finally started on my dissertation writing in earnest (and not grant writing), I needed to make sure that footnotes are usable in my work flow. So this is a quick write up of the tools I will use in writing my dissertation, and how I will use them.

The Tools

LibreOffice: Free and Open Source document software. Who knows how long I will have access to free Microsoft Word? LibreOffice (the fork of OpenOffice) will always be free and freely available. The steps will be basically the same if you are using Microsoft Word, just substitute that program for LibreOffice when it comes to it.

Zotero: I’m certainly biased, but Zotero is the greatest citation management software evar! Also free and open source. I’m using the stand alone version, but you can use the Firefox extension as well. Should work the same.

Scrivener: The greatest writing software I’ve seen. So good I even paid for it. I don’t usually do that with software (as you can see, I like free and open source).

The Process

Here I will try to outline the process I found that will save footnotes from existing documents into Scrivener, and Scrivener created footnotes into exported documents. From there, it’s easy to create Zotero connected footnotes.

1. Copy existing documents with footnotes into Scrivener

Copy from LibreOffice

The first issue to run across is to put your existing documents into scrivener. I wrote a paper for Hist 811 that is basically the bulk of Chapter 1 and Chapter 2 of the dissertation. It’s needs some finessing in order to fit in the dissertation. It would be a shame to lose the footnotes, which is what happens if you just use Scrivener’s import file process. This is an easy fix. Just copy the text from your document and paste it into a Scrivener text area.

Then with your Scrivener project open, create a new text area, or select an existing one, which ever, and paste it in. Nothing special there.

2. Create new footnotes in Scrivener

See how Scrivener makes a footnote!

What is special, though is what Scrivener does with that footnote. See there, footnote number 20, right after the quote about the cocktail of causes and rearmament being one of the ingredients? Now in Scrivener we have the word “ingredient” highlighted and underlined, and on the right side of the Scrivener window, there is a new footnote with all of the content of the original footnote. Sweet!

Easy as Format->Footnote, or use the shorcut keys Ctrl-Cmd-8

That’s all well and good. What if we want to edit the text a little bit, add some good stuff and add another footnote in there? What do we do? Well, Scrivener has a way to add a footnote. Just highlight some text (the footnote will be inserted after the last word), and go to the Format menu and select Footnote. Or you can use the fancy shortcut keys, for faster typing and footnote inserting, Ctrl-Cmd-8 (⌃⌘8).

Look, Ma! A new footnote!

Now you have a new, blank, footnote area to put a footnote reverence in.

Select the reference in Zotero and drag it into the footnote box in Scrivener.

Zotero makes it easy to put the reference in that new empty footnote with drag and drop citations. Just pull up your Zotero (either from Firefox, or if you have the standalone version). Select the reference you want, and drag it into the empty footnote section.

3. Moving from Scrivener to a document, and keeping your footnotes!

So, ideally, you would be able to export your text document, and all of these lovely footnotes you have made in Scrivener, using Zotero, would just magically work in a Word or LibreOffice document. It doesn’t, yet (or ever?). So here is how to get your footnotes into a document, and then get those footnotes to be Zotero enabled.

Srivener->File->Export->Files

First, you export your Scrivener document to RTF format.

Select RTF format

Select the plain RTF format, and the first check box for only the selected files (although, you could un check this if you want to do all of your files at once. No other check boxes are needed. Then just hit the Export button.

Open it up with your favorite document program, LibreOffice or Word.

Next, you will want to open your new RTF document in LibreOffice (or Word if you’re using that program).

All my citations are in the house!

You will notice that all of your footnotes are in this file. Yeah! Sometimes the text had odd font sizes and styles. So a quick ‘Select All’ and change it to default style and Times New Roman, 12 pt should fix that right up. Now here is the labor intensive part. For each footnote, we’re going to have to recreate it so that it is handled by Zotero. Then we’ll delete the original footnote. It would be nice of Scrivener could export the footnotes in a way that Zotero could detect them, but alas it is not to be.

Now you add a citation through the zotero buttons to make a zotero-aware citation.

All my citations are in the house!

Insert a Zotero citation using the Zotero buttons in your document program’s menu bar.

I prefer the Zotero classic view.

The new citation find view is pretty slick, though.

You can add pages with a coma, space, number.

Now you have two citations.

With two citations in the document, you’ll need to delete the one that was not made by zotero.

Just make sure you delete the non-zotero aware citation. The Zotero citation is usually highlighted.

Now you can save the document as a different file format: odt, doc, docx

Now save the document as an ODT document. If it is saved as anything else, it will not be Zotero aware.

Take your pick of file types.

Save as the correct file format if you want Zotero to be able to edit them again.

One alternative method is to create footnotes in Scrivener using the format {Author, Year, Page#}. Then export as an RTF document as before. Then, in Zotero, use the ‘RTF Scan’ tool in the Preferences menu. Zotero will see all of the citations and replace them nicely with formatted citations (using Ibid. and short notation for repeat books, and such). Zotero will not be aware of these citations at all, so if you need them to be Zotero aware, you might as well use the steps outlined above. If you do not expect to update citations or the text once done in Scrivener, then this may be the easiest way to go.

Now I can happily transfer existing documents into Scrivener and save the footnotes!