OVH: Custom Installation

 Back to Category:Arch Linux

Introduction

I have been running my own dedicated server for about 20 years now. I started running FreeBSD servers, had a couple OpenBSD servers even, but some time around early 2013 I made the switch to using Arch Linux. Have I gone mad? Possibly, but that's not relevant. Some might say Arch Linux does not belong on a server, or in a production environment. Nonsense! Any Linux distribution, properly setup, can be run as a server in a production environment. I use Arch on my desktop and on my laptop and honestly the setup on my servers does not diff too much from those installations aside from a little redundancy. I apply the same security principles and practices to both.

For the creation of this tutorial I used my server with OVH's subsidiary So you Start (SYS). The process is the same for OVH and Kimsufi (aside from the Manager being a bit different), I have had servers with all three.

Installation

First in order to install Arch Linux from scratch we need to get the server net-booted into Rescue Mode. However, before you do so it would be a good idea to note down a few things from one of the OVH default installations (doesn't matter which OS as long as it is Linux based). Take a look at the current network configuration, for their Arch Linux installation you can check netctl.

# cat /etc/netctl/ovh_net_eth0

Also check which timezone the datacenter is in if you are not using the BHS datacenter (the one I am).

# ls -FlG /etc/localtime

Alternatively if you installed the OS with the OVH installer through the Manager you can just check the file left behind by the installer.

# cat /root/.ovhrc

Rescue Mode

Enter the Control Panel and reboot the machine into Rescue Mode. There should be either a button on the right called "Rescue" or "Netboot".

OVH will email you the root password for rescue mode ssh login, make sure your spam filter did not catch it.

Once the machine has been restarted into rescue mode (it might require a manual restart) and login information has been obtained, login over ssh. These additional command line options here will make it so it does not record the hostkey, given that this is about to change once Arch is installed.

# ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no root@X.X.X.X

WARNING: The moment you login change the root password with passwd, OVH emails passwords in plain-text which is susceptible to MITM attacks.

Take note of the Debian environment.

Partitioning

For most of the mid to higher tier servers you get two hard drives. Using RAID-10 (far 2 layout), which prioritizes read speed, availability and redundancy will be optimal. Since there is only the two drives, write speeds should be on par with RAID1. LVM can then be used on top of RAID for partition management.
Pull up a list of all of the disks in the system.

# lsblk

If the server was previously installed in a non-RAID setup, it should be mounted inside of /mnt. If so, un-mount the partition.

# umount /mnt/sda1

Wipe the current disks so there is no interference.

# dd if=/dev/zero of=/dev/sda bs=1M count=2048 # dd if=/dev/zero of=/dev/sdb bs=1M count=2048

If there was a RAID volume previously on the drives, a reboot may be required.

Wipe the current partition scheme and create a new partition table.

# echo -e "o\nn\np\n\n\n+64G\na\nn\np\n\n\n+8G\nn\np\n\n\n\nw" | fdisk /dev/sda

Then copy the partition table to the second drive.

# sfdisk -d /dev/sda | sfdisk /dev/sdb

This will setup a 64GB root partition with the boot flag set, an 8GB swap partition and another data partition with the rest of each disk.
The hostname gets prefixed onto the RAID arrays name, might be a good idea to change it.

# hostname neutron

WARNING: If your do not set the hostname the RAID arrays name will be prefixed with rescue.ovh.net

RAID10 (far2)

Create both RAID10 arrays and add the disks.

# mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=2 /dev/sd[ab]1 # mdadm --create /dev/md1 --level=10 --layout=f2 --raid-devices=2 /dev/sd[ab]2

It will take a long time to synchronize the array. Check the status with watch -n .1 cat /proc/mdstat Synchronization is transparent to the file-system, continue installing and reboot if needed. It will be using significant disk resources until then.

When synchronization is finished, feel free to check to see if any blocks were marked bad.

# cat /sys/block/md0/md/mismatch_cnt

LVM

Create the LVM physical volume and put the RAID array inside, then verify it was added.

# pvcreate /dev/md0 # pvdisplay

Create a volume group and put the RAID array inside, then it was added.

# vgcreate neutron /dev/md0 # vgdisplay

Dedicate the entire array to the root logical volume, then verify.

# lvcreate -l +100%FREE neutron -n root # lvdisplay

This can be re-sized at any point but due to the partition size cannot exceed 64GB.

Formatting

Format the root filesystem.

# mkfs.ext4 /dev/neutron/root

Be sure to select Y to overwrite the current partition if asked.

Now that the disks are setup, remove the directory OVH created in /mnt if applicable and then mount the logical volume.

# rmdir /mnt/* # mount /dev/neutron/root /mnt

Bootstrap

At this point download the arch-bootstrap.sh script that was created by Arch Linux user tokland. Then give the script the proper permissions.

# wget https://raw.githubusercontent.com/tokland/arch-bootstrap/master/arch-bootstrap.sh # chmod +x arch-bootstrap.sh

GitHub: tokland/arch-bootstrap

Then bootstrap the system (ignore the tar errors).

# ./arch-bootstrap.sh /mnt

Next mount the required system partitions. Don't forget the last one added here.

# mount -t proc proc /mnt/proc/ # mount -t sysfs sys /mnt/sys/ # mount -o bind /dev /mnt/dev/

Chroot

Chroot into the new install.

# chroot /mnt

Initialize and populate the Arch Linux pacman keys.

# pacman-key --init # pacman-key --populate archlinux

Finally re-base the system by running pacman to install Arch Linux.

# pacman -S base base-devel arch-install-scripts git grub-bios intel-ucode openssh sudo vim

Update the RAID configuration to reflect what was setup.

# mdadm --examine --scan > /etc/mdadm.conf

Generate an fstab file.

# genfstab / > /etc/fstab

Changes will need to be made to the /etc/mkinitcpio.conf for RAID and LVM, open up the file in an editor.

filename: /etc/mkinitcpio.conf

MODULES="dm_mod" HOOKS="base udev mdadm_udev lvm2 autodetect modconf block filesystems keyboard fsck"

Then generate a new initramfs.

# mkinitcpio -p linux

Boot Loader

Use the scripts that come with grub in order to install the boot loader on the first disk.

# grub-install /dev/sdX

Generate the GRUB config.

# grub-mkconfig -o /boot/grub/grub.cfg

Networking

Next create a system service to start up the network.

# vim /etc/systemd/network/eth0.network

[Match] Name=eth0 en* [Network] DHCP=false DNS=213.186.33.99 Address=X.X.X.X/24 Address=X.X.X.X/24 Gateway=X.X.X.X Address=ffff:ffff:ffff:ffff::/64 Gateway=ffff:ffff:ffff:ffff

Multiple IPv4 Address lines are included in-case one purchased additional failover IP addresses.

Make sure the naming scheme stays how OVH had it setup so the device node is named the same.

# cd /etc/udev/rules.d # touch 80-net-name-slot.rules # ln -s /dev/null 80-net-setup-link.rules

Then enable systemd-networkd so that the service is started on boot, also enable the ssh service.

# systemctl enable systemd-networkd # systemctl enable sshd

System Settings

Choose a unique hostname for the machine.

# echo "neutron.kyau.net" > /etc/hostname

Set the proper timezone and generate /etc/adjtime. Check the /usr/share/zoneinfo directory for a list of existing zones.

My VPS is in the BHS datacenter so I will use the timezone that applies.

# ln -sf /usr/share/zoneinfo/America/Toronto /etc/localtime

Choose the proper localization and uncomment it from /etc/locale.gen, the defacto English (US) standard is en_US.UTF-8 UTF-8, and then generate the needed localization.
Also add the localization to /etc/locale.conf.

# sed -i 's/^#en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen # locale-gen # echo "LANG=en_US.UTF-8" > /etc/locale.conf

Set a root password.

# passwd

Create a user account adding it to the wheel group, then give it a password.

# useradd -m -G wheel,systemd-journal -s /bin/bash kyau # passwd kyau

Make sure the wheel group has sudo access.

# sed -i 's/^# %wheel ALL=(ALL) ALL$/%wheel ALL=(ALL) ALL/' /etc/sudoers

Reboot

Exit out of the chroot environment.

# exit

Navigate back to the Web UI for the OVH VPS and click on the button on the right labeled "Reboot my VPS", this will pull the server back out of rescue mode.

Post-Installation

Once the machine has rebooted and you can see a green light on PING and SSH in the OVH dashboard, ssh back into your machine.

# ssh kyau@neutron.kyau.net

Network Time Protocol

As it is always a good idea to keep the systems date/time in sync, for this use systemd-timesyncd.

# sudo timedatectl set-ntp true

Ext4 Metadata Checksums

Since the Debian rescue environment did not have support for metadata checksums, it had to be postponed until after the reboot into Arch.
Check and optimize the root partition.

# sudo e2fsck -Df /dev/neutron/root

The filesystem then needs to be converted to 64-bit.

# sudo resize2fs -b /dev/neutron/root

Then checksums can be added to the partition.

# sudo tune2fs -O metadata_csum /dev/neutron/root

Verify they were added to the partition.

# sudo dumpe2fs -h /dev/neutron/root

Add the required modules (including the Intel modules) to your initramfs.

filename: /etc/mkinitcpio.conf

MODULES="dm_mod crc32_generic crc32c-intel"

Then modify the fstab file to also add journal checksums.

filename: /etc/fstab

/dev/mapper/neutron-root / ext4 rw,relatime,stripe=256,data=ordered,journal_checksum 0 1

Packages

Moving the build directory that makepkg uses to build packages into the /tmp folder will effectively move the package building temporary directory into memory.

filename: /etc/makepkg.conf

BUILDDIR=/tmp/makepkg

To find the fastest recently updated mirrors for pacman, install the reflector package.

# sudo pacman -S reflector

Reflector will search through the last 100 updated mirrors that use HTTPS and sort them by download speed and then output the list to /etc/pacman.d/mirrorlist overwriting the current list. This command might take a few minutes depending on internet connection speed and latency.

# sudo reflector --latest 100 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

Enable colored output in pacman.

# sudo sed -i 's/#Color/Color/' /etc/pacman.conf

To install aur:pacaur first manually install cower.

# git clone https://aur.archlinux.org/cower.git # cd cower && makepkg -si && cd .. && rm -rf cower

If makepkg complains about GPG keys see this pinned comment or use gpg --recv-keys --keyserver hkp://pgp.mit.edu 1EB2638FF56C0C53

Then proceed to install pacaur manually.

# git clone https://aur.archlinux.org/pacaur.git # cd pacaur && makepkg -si && cd .. && rm -rf pacaur

Perform a full system update.

# pacaur -Syu

For more information on managing and maintaining packages you can reference my tutorial on packages.

Encryption

For my data partition I prefer to use LVM on LUKS instead of LUKS on LVM (this is why there were two partitions created) as it has the added advantage of unlocking all LVM volumes with a single key and being able to change around the LVM without messing with encryption.

Security

Security is always of paramount concern when running a machine that has open access to the internet access. While I am not a security professional I do have some practices I like to follow. Every Arch machine I run that sits with an open internet connection (server, production, or otherwise) gets hardened. I have been doing this with all of my machines since my days with BSD. That said, I am always on the look out for new tips and tricks. Have any?

See also: Hardening Arch Linux 

Finalizing

Once hardening is complete, the machine should be setup with LVM on top of RAID10(far2) with an Arch Linux boot partition sized to 64GB. The rest of the space available in the volume group available for encrypted partitions, containers, backups or whatever you wish.

If you would like to install services (DNS, WWW, SQL, etc.) inside of containers see the article on systemd-nspawn.

See also: Containers with systemd-nspawn