Server Migration – Update

Following the early June Server Farm fire and subsequent down time here is a follow up on the progress so far and short terms plans.

The new server in the UK is nearly ready for production and to become the primary server, this will have a impact on server response as there are fewer nodes to go through to reach UK/EU customer base.

However this will be very minimal gains and US, rest of the world visitors will not see a difference. I will also operate a fail over policy for dns, emails and websites:

All websites will be synchronised on an daily basis between the 2 servers, in case of lasting downtime from the UK server, we will be able to reverse to the US server within hours (the time it will take to manually change the DNS records)

For email service, the UK server will handle emails. However, as email is a bit more resilient than websites in a way that if emails cannot be delivered they will be queued until they can, a fail over is less critical.

I am currently experimenting with the possibility of having the two servers able to receive the same email message, if one is not working, the other should. This would require having two email accounts for the same address and access these two accounts using the IP address of the mail server instead of the domain name, it is fairly technical and needs to be tested for feasibility first.

As for DNS, I will also have more than one name server (ie phone book) as DNS protocol require at least one of these to minimise interruption, I will have the record stored in 6 name servers located in different locations. The websites will be unreachable only if the Internet goes entirely down.

Sharpnet.info

I am planning to have the sharpnet network information on two different servers, one in The UK (uk.sharpnet.info) and in the US (us.sharpnet.info) that way if one server is down the other will contain the reason of the problem.

Passwordless SSH

Using encrypted keys for authentication offers two main benefits. Firstly, it is convenient as you no longer need to enter a password (unless you encrypt your keys with password protection) if you use public/private keys. Secondly, once public/private key pair authentication has been set up on the server, you can disable password authentication completely meaning that without an authorized key you can’t gain access – so no more password cracking attempts.

It’s a relatively simple process to create a public/private key pair and install them for use on your ssh server.

First, create a public/private key pair on the client that you will use to connect to the server (you will need to do this from each client machine from which you connect):

ssh-keygen -t rsa

This will create two files in your (hidden) ~/.ssh directory called id_rsa and id_rsa.pub. id_rsa is your private key and id_rsa.pub is your public key.

If you don’t want to still be asked for a password each time you connect, just press enter when asked for a password when creating the key pair. It is up to you to decide whether or not you should password encrypt your key when you create it. If you don’t password encrypt your key, then anyone gaining access to your local machine will automatically have ssh access to the remote server. Also, root on the local machine has access to your keys although one assumes that if you can’t trust root (or root is compromised) then you’re in real trouble. Encrypting the key adds additional security at the expense of eliminating the need for entering a password for the ssh server only to be replaced with entering a password for the use of the key.

Now set permissions on your private key:

chmod 700 ~/.ssh ; chmod 600 ~/.ssh/id_rsa

Copy the public key (id_rsa.pub) to the server and install it to the authorized_keys list (or open the authorized_keys list file and paste the content of id_rsa.pub into it) :

cat id_rsa.pub >> ~/.ssh/authorized_keys

Note: once you’ve imported the public key, you can delete it from the server.

and finally set file permissions on the server:

chmod 700 ~/.ssh ; chmod 600 ~/.ssh/authorized_keys

The above permissions are required if StrictModes is set to yes in /etc/ssh/sshd_config (the default).

Disaster Recovery Plan

A complete memo of Sharpnet UK disaster recovery strategy will be laid out here. This will prove a valuable source of information to other Dedicated server administrators, especially new players.

Comments welcome!

Plesk Migration Manager (PMM)

Just to share my experience with this application. If you are migrating Plesk (8.x) to Plesk (8.x) (ie another server running plesk) and you have little space left on the /home (or /) hard disk partition on the source server.

You must also have a partition that has at least enough space to hold the /var/www/vhost content…

Plesk migration manager stores dumped file under /home/migration. To alleviate the risk of filling the / or /home partition do this:

rmdir /home/migration
ln -s {otherpartitiondir} /home/migration

where {otherpartitiondir} is a full path on the partition that has enough space.

Plesk is not really telling you where the stuff is stored (or I did not find it)

The Planet DC H1 Crash

The Planet Houston Datacenter 1 (US) or Server Farm 1 (GB) electrical power equipment blew last Saturday and this resulted in a catastrophic service interruption that is still felt today 🙁 we still face intermittent service but it is getting better.

The consequences of this accident, where no one was hurt, has made me rethink the security and vulnerability of the server we have there. We now are in the process of migrating to a new server in the UK and this should take a couple of days.