OVH: Custom Installation

 Back to Category:Arch Linux

Introduction

I have been running my own dedicated server for about 20 years now. I started running FreeBSD servers, had a couple OpenBSD servers even, but some time around early 2013 I made the switch to using Arch Linux. Have I gone mad? Possibly, but that's not relevant. Some might say Arch Linux does not belong on a server, or in a production environment. Nonsense! Any Linux distribution, properly setup, can be run as a server in a production environment. I use Arch on my desktop and on my laptop and honestly the setup on my servers does not diff too much from those installations aside from a little redundancy. I apply the same security principles and practices to both.

For the creation of this tutorial I used my server with OVH's subsidiary So you Start (SYS). The process is the same for OVH and Kimsufi (aside from the Manager being a bit different), I have had servers with all three.

Installation

First in order to install Arch Linux from scratch we need to get the server net-booted into Rescue Mode. However, before you do so it would be a good idea to note down a few things from one of the OVH default installations (doesn't matter which OS as long as it is Linux based). Take a look at the current network configuration, for their Arch Linux installation you can check netctl.

# cat /etc/netctl/ovh_net_eth0

Also check which timezone the datacenter is in if you are not using the BHS datacenter (the one I am).

# ls -FlG /etc/localtime

Alternatively if you installed the OS with the OVH installer through the Manager you can just check the file left behind by the installer.

# cat /root/.ovhrc

Rescue Mode

Enter the Control Panel and reboot the machine into Rescue Mode. There should be either a button on the right called "Rescue" or "Netboot".

OVH will email you the root password for rescue mode SSH login, make sure your spam filter did not catch it.

Once the machine has been restarted into rescue mode (it might require a manual restart) and login information has been obtained, login over SSH. These additional command line options here will make it so it does not record the hostkey, given that this is about to change once Arch is installed.

# ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no root@X.X.X.X

WARNING: The moment you login change the root password with passwd, OVH emails passwords in plain-text which is susceptible to MITM attacks.

Take note of the Debian environment.

Partitioning

For most of the mid to higher tier servers you get two hard drives. Using RAID-10 (far 2 layout), which prioritizes read speed, availability and redundancy will be optimal. Since there is only the two drives, write speeds should be on par with RAID1. LVM can then be used on top of RAID for partition management.
Pull up a list of all of the disks in the system.

# lsblk

If the server was previously installed in a non-RAID setup, it should be mounted inside of /mnt. If so, un-mount the partition.

# umount /mnt/sda1

Wipe the current disks so there is no interference.

# dd if=/dev/zero of=/dev/sda bs=1M count=2048 # dd if=/dev/zero of=/dev/sdb bs=1M count=2048

If there was a RAID volume previously on the drives, a reboot may be required.

Wipe the current partition scheme and create a new partition table.

# echo -e "o\nn\np\n\n\n+64G\na\nn\np\n\n\n+8G\nn\np\n\n\n\nw" | fdisk /dev/sda

Then copy the partition table to the second drive.

# sfdisk -d /dev/sda | sfdisk /dev/sdb

This will setup a 64GB root partition with the boot flag set, an 8GB swap partition and another data partition with the rest of each disk.
The hostname gets prefixed onto the RAID arrays name, might be a good idea to change it.

# hostname neutron

WARNING: If your do not set the hostname the RAID arrays name will be prefixed with rescue.ovh.net

RAID10 (far2)

Create both RAID10 arrays and add the disks.

# mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=2 /dev/sd[ab]1 # mdadm --create /dev/md1 --level=10 --layout=f2 --raid-devices=2 /dev/sd[ab]3

It will take a long time to synchronize the array, synchronization is transparent to the file-system, continue installing and reboot if needed. It will be using significant disk resources until then. To check the status of the synchronization.

# watch -n .1 cat /proc/mdstat

When synchronization is finished, feel free to check to see if any blocks were marked bad.

# cat /sys/block/md0/md/mismatch_cnt

During the rest of the installation due to synchronization things involving grub may take longer than normal. Also, if you see Failed to connect to lvmetad during install, ignore it as it is related to sync as well.

LVM

Create the LVM physical volume and put the RAID array inside, then verify it was added.

# pvcreate /dev/md0 # pvdisplay

Create a volume group and put the RAID array inside, then it was added.

# vgcreate neutron /dev/md0 # vgdisplay

Dedicate the entire array to the root logical volume, then verify.

# lvcreate -l +100%FREE neutron -n root # lvdisplay

This can be re-sized at any point but due to the partition size cannot exceed 64GB.

Also prepare the second RAID array as a volume group for later usage.

# pvcreate /dev/md1 # vgcreate data /dev/md1

Formatting

Formatting the root filesystem would be simple if we did not want metadata checksums. The Debian environment I was in when I performed my installation had an older version of e2fsprogs. In order to use metadata checksums you need version 1.43+, therefore it had to be upgraded first.
Begin by updating the package registry.

# apt-get update

Check which version of e2fsprogs is installed and which are available.

# apt-get policy e2fslibs

Pick a version that is 1.43+ if not already installed. Don't forget to upgrade both packages.

# apt-get install -y e2fslibs=1.43.3-1~bpo8+1 e2fsprogs=1.43.3-1~bpo8+1

Format the filesystem with metadata_csum and 64bit flags enabled.

# mkfs.ext4 -O metadata_csum,64bit /dev/neutron/root

Be sure to select Y to overwrite the current partition if asked.

Now that the disks are setup, remove the directory OVH created in /mnt if applicable and then mount the logical volume.

# rmdir /mnt/* # mount /dev/neutron/root /mnt

Bootstrap

At this point download the arch-bootstrap.sh script that was created by Arch Linux user tokland. Then give the script the proper permissions.

# wget https://raw.githubusercontent.com/tokland/arch-bootstrap/master/arch-bootstrap.sh # chmod +x arch-bootstrap.sh

GitHub: tokland/arch-bootstrap

Then bootstrap the system (ignore the tar errors).

# ./arch-bootstrap.sh /mnt

Next mount the required system partitions. Don't forget the last one added here.

# mount -t proc proc /mnt/proc/ # mount -t sysfs sys /mnt/sys/ # mount -o bind /dev /mnt/dev/

Chroot

Chroot into the new install.

# chroot /mnt

Initialize and populate the Arch Linux pacman keys.

# pacman-key --init # pacman-key --populate archlinux

Finally re-base the system by running pacman to install Arch Linux.

# pacman -S base base-devel arch-install-scripts git grub-bios intel-ucode openssh sudo vim

Parts of this step, most notably grub generating its config, took about 5 minutes. This is due to the half-setup environment and RAID arrays initializing, trust me its working fine just wait it out.

Update the RAID configuration to reflect what was setup.

# mdadm --examine --scan > /etc/mdadm.conf

Generate an fstab file.

# genfstab / > /etc/fstab

Open up the fstab file in an editor and add the flags for metadata checksums to the root partition.

filename: /etc/fstab

/dev/mapper/neutron-root / ext4 rw,relatime,stripe=256,data=ordered,journal_checksum 0 1

Changes will need to be made to the /etc/mkinitcpio.conf for RAID and LVM and Ext4 metadata checksums, open up the file in an editor.

filename: /etc/mkinitcpio.conf

MODULES="dm_mod crc32_generic crc32c-intel" HOOKS="base udev mdadm_udev lvm2 autodetect modconf block filesystems keyboard fsck"

Then generate a new initramfs.

# mkinitcpio -p linux

Boot Loader

Use the scripts that come with grub in order to install the boot loader on the first disk.

# grub-install /dev/sda

Generate the GRUB config.

# grub-mkconfig -o /boot/grub/grub.cfg

WARNING: Due to RAID array synchronization these steps might take an abnormally long time

Networking

Next create a system service to start up the network.

# vim /etc/systemd/network/eth0.network

[Match] Name=eth0 en* [Network] DHCP=false DNS=213.186.33.99 Address=X.X.X.X/24 Address=X.X.X.X/24 Gateway=X.X.X.X Address=ffff:ffff:ffff:ffff::/64 Gateway=ffff:ffff:ffff:ffff

Multiple IPv4 Address lines are included in-case one purchased additional failover IP addresses.

Make sure the naming scheme stays how OVH had it setup so the device node is named the same.

# cd /etc/udev/rules.d # touch 80-net-name-slot.rules # ln -s /dev/null 80-net-setup-link.rules

Then enable systemd-networkd so that the service is started on boot, also enable the SSH daemon.

# systemctl enable systemd-networkd # systemctl enable sshd

System Settings

Choose a unique hostname for the machine.

# echo "neutron.kyau.net" > /etc/hostname

Set the proper timezone and generate /etc/adjtime. Check the /usr/share/zoneinfo directory for a list of existing zones.

My server is in the BHS datacenter so I will use the timezone that applies.

# ln -sf /usr/share/zoneinfo/America/Toronto /etc/localtime

Choose the proper localization and uncomment it from /etc/locale.gen, the defacto English (US) standard is en_US.UTF-8 UTF-8, and then generate the needed localization.
Also add the localization to /etc/locale.conf.

# sed -i 's/^#en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen # locale-gen # echo "LANG=en_US.UTF-8" > /etc/locale.conf

Set a root password.

# passwd

Create a user account adding it to the wheel group, then give it a password.

# useradd -m -G wheel,systemd-journal -s /bin/bash kyau # passwd kyau

Make sure the wheel group has sudo access.

# sed -i 's/^# %wheel ALL=(ALL) ALL$/%wheel ALL=(ALL) ALL/' /etc/sudoers

Reboot

Exit out of the chroot environment.

# exit

Navigate back to the Web Manager for the OVH server and put netboot back into local hard drive boot mode.
Un-mount all of the partitions and reboot.

# umount /mnt/{dev,proc,sys} /mnt # reboot

Post-Installation

Once the machine has rebooted, SSH back into your machine.

# ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no kyau@neutron.kyau.net

First verify that the root partition was setup properly with metadata checksums and running in 64-bit mode.

# sudo dumpe2fs -h /dev/neutron/root | grep -e checksum -e features

Network Time Protocol

As it is always a good idea to keep the systems date/time in sync, for this use systemd-timesyncd.

# sudo timedatectl set-ntp true

Packages

Moving the build directory that makepkg uses to build packages into the /tmp folder will effectively move the package building temporary directory into memory.

filename: /etc/makepkg.conf

BUILDDIR=/tmp/makepkg

To find the fastest recently updated mirrors for pacman, install the reflector package.

# sudo pacman -S reflector

Reflector will search through the last 100 updated mirrors that use HTTPS and sort them by download speed and then output the list to /etc/pacman.d/mirrorlist overwriting the current list. This command might take a few minutes depending on internet connection speed and latency.

# sudo reflector --latest 100 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

Enable colored output in pacman.

# sudo sed -i 's/#Color/Color/' /etc/pacman.conf

To install aur:pacaur first manually install cower.

# git clone https://aur.archlinux.org/cower.git # cd cower && makepkg -si && cd .. && rm -rf cower

If makepkg complains about GPG keys see this pinned comment or use gpg --recv-keys --keyserver hkp://pgp.mit.edu 1EB2638FF56C0C53

Then proceed to install pacaur manually.

# git clone https://aur.archlinux.org/pacaur.git # cd pacaur && makepkg -si && cd .. && rm -rf pacaur

Perform a full system update.

# pacaur -Syu

For more information on managing and maintaining packages you can reference my tutorial on packages.

Security

Security is always of paramount concern when running a machine that has open access to the internet access. While I am not a security professional I do have some practices I like to follow. Every Arch machine I run that sits with an open internet connection (server, production, or otherwise) gets hardened. I have been doing this with all of my machines since my days with BSD. That said, I am always on the look out for new tips and tricks. Have any?

See also: Hardening Arch Linux 

Finalizing

Once hardening is complete, the machine should be setup with LVM on top of RAID10(far2) with an Arch Linux boot volume sized to 64GB and the rest of the disk space in a separate volume group ready for logical volume creation however you see fit.

If you would like to install services (DNS, WWW, SQL, etc.) it is recommended to do so using either KVM or containers to separate them from the host system.

See also: KVM on Arch Linux