FreeBSD:ZFS Install: Difference between revisions

From Wiki³
mNo edit summary
mNo edit summary
 
(47 intermediate revisions by the same user not shown)
Line 1: Line 1:
The following is how to install FreeBSD on ZFS root using GPT. This will offer redundancy and scaling of performance (with additional drives) for you to base your system off of while maintaining efficiency of disk usage. This guide is mainly suited for workstations, home file servers, and media center PCs. It is setup for use with FreeBSD 9+ and was directly tested with 9.0-CURRENT-201101-amd64 version of FreeBSD with the new BSDInstall<ref>{{cite web ||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=218799|title=FreeBSD Base r218799, Fri Feb 18 2011}}</ref>.
{{DISPLAYTITLE:FreeBSD ZFS Installation}}
<div id="tocalign">__TOC__</div>
The following is how to install FreeBSD on ZFS root using GPT. This will offer redundancy and scaling of performance (with additional drives) for you to base your system off of while maintaining efficiency of disk usage. This guide is mainly suited for workstations, home file servers, and media center PCs. It is setup for use with FreeBSD 9+ and was directly tested with 9.0-RELEASE version of FreeBSD which includes the new BSDInstall<ref>{{cite web ||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=218799 |title=FreeBSD Base r218799, Fri Feb 18 2011}}</ref>.


==Introduction to FreeBSD==
==Introduction to ZFS==
I am going to make the assumption that you have at least a basic knowledge of the command line in FreeBSD. If you are new to FreeBSD all hope is not lost. The [http://www.freebsd.org/doc/handbook/ FreeBSD Handbook] is the primary source of documentation produced by the FreeBSD Documentation Project. If you are familiar with linux but new to FreeBSD there is the [http://www.freebsd.org/doc/en/articles/linux-users/index.html FreeBSD Quickstart Guide for Linux Users]. Last but not least if you are coming over from Windows you can take a look at [http://vtbsd.net/notwindows.html FreeBSD is NOT Windows]. (Recently there has also been another great article written by Paul Venezia at InfoWorld entitled [http://www.infoworld.com/d/data-center/why-arent-you-using-freebsd-178119 Why aren't you using FreeBSD?])
ZFS became part of FreeBSD on 6th April 2007, while it stayed in the experimental phase until the 15th of September 2009<ref>{{cite web ||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=197221 |title=FreeBSD Base r197221, Tue Sep 15 2009}}</ref> when they removed the experimental tag and deemed it ready for production use. Then on the 23th of November 2009<ref>{{cite web ||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=199714 |title=FreeBSD Base r199714, Mon Nov 23 2009}}</ref> zfsloader was merged into -STABLE making ZFS as boot option easier than ever.


ZFS became part of FreeBSD on 6th April 2007, while it stayed in the experimental phase until the 15th of September 2009<ref>{{cite web||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=197221|title=FreeBSD Base r197221, Tue Sep 15 2009}}</ref> when they removed the experimental tag and deemed it ready for production use. Then on the 23th of November 2009<ref>{{cite web||url=http://svn.freebsd.org/viewvc/base?view=revision&revision=199714|title=FreeBSD Base r199714, Mon Nov 23 2009}}</ref> zfsloader was merged into -STABLE making ZFS as boot option easier than ever.
ZFS and other newer features that have been recently added into FreeBSD are not available through the generic {{ManPage|man=sysinstall|section=8}} that is included on all of the default installation media. To remedy this, I am going to show you how to install FreeBSD manually from the Fixit livefs command line.


geli is an easy to use block device encryption to encrypt disks on a lower level than the filesystem. We will be taking advantage of geli for our swap and temporary directories.
==Booting the Installation Media==


ZFS, geli and other newer features that have been recently added into FreeBSD are not available through the generic {{ManPage|man=sysinstall|section=8}} that is included on all of the default installation media. To remedy this, I am going to show you how to install FreeBSD manually from the Fixit livefs command line.
At this point you should boot from your installation media. For use in creation of this article the following test environment was used.
{{Note|icon=false|1={{Icon|computer}} VirtualBox 4.0.14_OSE r74382<br/>{{Icon|disc-blue}} FreeBSD-10.0-HEAD-20120120-JPSNAP-amd64-amd64-release.iso}}


==Boot the Installation Media==
When you first boot up your chosen media you will see the normal boot menu, feel free to wait out the allotted time or press enter. After the system is done booting up you will be presented with the 'Welcome' window inside the new BSDInstaller.
The first thing you are going to need is a copy of the most current {{mono|*-dvd1.iso}} or {{mono|*-memstick.img}} FreeBSD image (the other images do not contain the livefs that is needed). FreeBSD may be obtained from [http://torrents.freebsd.org:8080/ torrent] or an [http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/mirrors-ftp.html FTP mirror] near you.


At this point you should boot from your installation media. After the boot media initializes the loader you will be presented with the boot loader menu. Press {{mono|6}} to escape to the loader prompt, load the ahci module and continue to boot.
==Configuring the Installation Environment==
Select {{Mono|&lt; Shell &gt;}} from the options and press enter. You should now be presented with the shell. First thing we need to do is switch {{Mono|/tmp}} from read-only to writable (used later to store the zpool.cache file) and then load the required kernel modules via {{ManPage|man=kldload|section=8}}.


[[File:Freebsd_zfs_install-1.png|frame|FreeBSD Boot Loader Menu]]
{{Code|umount /dev/md1}}
 
Ignore any errors you may or may not see from executing this command, the {{Mono|/tmp}} directory will still be unmounted. Remount the tmp directory and load the needed kernel modules.
 
{{Code|mdmfs -s 512M md1 /tmp}}
 
{{Code|kldload geom_mirror}}
 
{{Code|kldload opensolaris}}
 
{{Code|kldload zfs}}
 
===Remote Installation Setup (Optional)===
 
This section takes two things for granted; (a) that you have a personal set of SSH keys, and (b) that they are located on another server on the network, one that is already running an SCP capable SSH server. If you do not have one of these requirements feel free to skip this section or pause to set this up on your own. I feel it is easier to install via SSH due to the large amount of actual typing you are required to do, whereas with SSH you can merely copy/paste into the SSH window.
 
Connect to the network via DHCP with {{ManPage|man=dhclient|section=8}} (for network interface name use {{ManPage|man=ifconfig|section=8}}), then create our SSH host keys via {{ManPage|man=ssh-keygen|section=1}}.
 
{{Code|dhclient em0}}
 
{{Code|cd /tmp}}
 
{{Code|<nowiki>ssh-keygen -f /tmp/hostkey -t rsa1 -b 4096 -N ''</nowiki>}}
 
{{Code|<nowiki>ssh-keygen -f /tmp/hostkey_dsa -t dsa -N ''</nowiki>}}
 
Next we will need to download your personal SSH keys from the network. Replace the username, IP address, and folder/file with the required information. After that is done go ahead and start up the SSH daemon with {{ManPage|man=sshd|section=8}}.
 
{{Code|scp user@192.168.''x''.''x'':~/.ssh/authorized_keys .}}
 
{{Code|<nowiki>/usr/sbin/sshd -o PermitRootLogin=yes -o HostKey=/tmp/hostkey -o HostDSAKey=/tmp/hostkey_dsa -o AuthorizedKeysFile=/tmp/authorized_keys</nowiki>}}
 
Connect to the IP address that was assigned to the installation computer using {{ManPage|man=ssh|section=1}} with your personal SSH keys.<ref>{{Cite web ||url=http://www.chiark.greenend.org.uk/~sgtatham/putty/ |title=PuTTY |publisher=If you are not using a Linux or BSD derived operating system you can alternatively use PuTTY}}</ref>
 
{{Code|ssh -l root 192.168.''x''.''x''}}
 
==Disk Partition Setup==
Moving on to the disk setup, first locate all of your disks, we can use {{ManPage|man=dmesg|section=8}} and {{ManPage|man=grep|section=1}} for this. The system I am using for this guide has two disks, your results may vary.
 
{{Code|dmesg {{!}} grep -e "^ad" -e "^da" -e "^hd"}}
{{Output|<nowiki>ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 512000MB (1048576000 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 512000MB (1048576000 512 byte sectors: 16H 63S/T 163</nowiki>}}
 
For partition table setup we will be using GUID Partition Table (GPT) via {{ManPage|man=gpart|section=8}}. Let's first pull up a list of the current partitions.
 
{{Code|gpart show}}
 
If there are existing partitions on any disk remove them with the delete command, then destroy the disks partitioning scheme.
 
{{Code|gpart delete -i 1 ada0}}
 
{{Code|gpart destroy ada0}}
 
Create a new partitioning scheme on each disk.
 
{{Code|gpart create -s gpt ada0}}
 
{{Code|gpart create -s gpt ada1}}
 
Now we need to create all of the disk partitions, we will be creating four partitions on each disk. The first partitions will contain the boot loader, the second partitions are the swap partitions and finally the third partitions are for the zpool (ZFS Pool). Adjust the size of the zpool partition to fit your disk size needs.
 
{{Code|gpart add -s 64K -t freebsd-boot ada0}}
 
{{Code|gpart add -s 4G -t freebsd-swap -l swap0 ada0}}
 
{{Code|gpart add -s 495G -t freebsd-zfs -l zroot0 ada0}}
 
Now that we have the base partition setup, replicate this onto the other disk.
 
{{Code|gpart add -s 64K -t freebsd-boot ada1}}
 
{{Code|gpart add -s 4G -t freebsd-swap -l swap1 ada1}}
 
{{Code|gpart add -s 495G -t freebsd-zfs -l zroot1 ada1}}
 
Install the protected MBR boot code and gptzfsboot loader onto each of the disks. The gptzfsboot loader will give you the ability to load zfsloader from a zpool.
 
{{Code|gpart bootcode -b /boot/pmbr -i 1 -p /boot/gptzfsboot ada0}}
 
{{Code|gpart bootcode -b /boot/pmbr -i 1 -p /boot/gptzfsboot ada1}}
 
Verify the partition table to make sure all of the disks match.
 
{{Code|gpart show}}
{{Output|<nowiki>=>        34  1048575933  ada0  GPT  (500G)
          34        128    1  freebsd-boot  (64K)
        162    8388608    2  freebsd-swap  (4.0G)
    8388770  1038090240    3  freebsd-zfs  (495G)
  1046479010    2096957        - free -  (1.0G)
 
=>        34  1048575933  ada1  GPT  (500G)
          34        128    1  freebsd-boot  (64K)
        162    8388608    2  freebsd-swap  (4.0G)
    8388770  1038090240    3  freebsd-zfs  (495G)
  1046479010    2096957        - free -  (1.0G)</nowiki>}}
 
Take note that I did not use the entire disk, the reason behind this is that when using RAID and replacing problematic disks more than likely your new disk will have a slightly different disk geometry and as such might limit your ability to use it as a replacement disk if you cannot fit all the necessary partitions on the disk.
 
==ZFS File System==
Moving onto the ZFS volume setup, create the main {{ManPage|man=zpool|section=8}} and the root volume, then mount the root volume on to {{Mono|/mnt}}.
 
{{Code|<nowiki>zpool create -O atime=off -O canmount=off -O checksum=fletcher4 -O mountpoint=/mnt -O setuid=off zroot raidz gpt/zroot0 gpt/zroot1</nowiki>}}
 
{{Code|<nowiki>zfs create -o mountpoint=legacy -o setuid=on zroot/root</nowiki>}}
 
{{Code|mount -t zfs zroot/root /mnt}}
 
Create the {{Mono|/usr}} container volume and then all the sub-volumes.
 
{{Code|<nowiki>zfs create -o canmount=off zroot/usr</nowiki>}}
 
{{Code|zfs create zroot/usr/home}}
 
{{Code|<nowiki>zfs create -o compress=lzjb -o exec=on zroot/usr/obj</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=gzip zroot/usr/ports</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=off -o exec=off zroot/usr/ports/distfiles</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=off -o exec=off zroot/usr/ports/packages</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=gzip -o exec=off zroot/usr/src</nowiki>}}
 
{{Code|cd /mnt ; ln -s /usr/home home}}
 
Then create the {{Mono|/var}} container and subsequent sub-volumes.
 
{{Code|<nowiki>zfs create -o exec=off -o canmount=off zroot/var</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=lzjb zroot/var/audit</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=lzjb zroot/var/crash</nowiki>}}
 
{{Code|zfs create zroot/var/db}}
 
{{Code|<nowiki>zfs create -o compress=lzjb -o exec=on zroot/var/db/pkg</nowiki>}}
 
{{Code|zfs create zroot/var/empty}}
 
{{Code|<nowiki>zfs create -o compress=gzip zroot/var/log</nowiki>}}
 
{{Code|<nowiki>zfs create -o compress=gzip zroot/var/mail</nowiki>}}
 
{{Code|zfs create zroot/var/run}}
 
==FreeBSD Installation==
Now that the ZFS file system hierarchy is setup we can remount the ZFS pool, install the FreeBSD base system, and then copy the zpool.cache. When you import the zpool back into the system you might get errors about not being able to create mountpoints, this is rectified two commands later.
 
{{Code|cd}}
 
{{Code|zpool export zroot}}
 
{{Code|zpool import -o cachefile{{=}}/tmp/zpool.cache zroot}}
 
{{Code|mount -t zfs zroot/root /mnt}}
 
{{Code|zfs mount -a}}
 
 
{{Code|sh}}
 
{{Code|cd /usr/freebsd-dist}}
 
{{Code|export DESTDIR{{=}}/mnt}}
 
{{Code|for file in base.txz lib32.txz kernel.txz doc.txz;}}
 
{{Code|<nowiki>do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done</nowiki>}}
 
 
{{Code|cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache}}
 
{{Code|exit}}
 
Switch the {{Mono|zroot/var/empty}} zpool volume to read-only and then chroot into your newly installed system.
 
{{Code|zfs set readonly{{=}}on zroot/var/empty}}
 
{{Code|chroot /mnt}}
 
==FreeBSD Configuration==
Setup the network and necessary startup files so the zpool will be mounted automatically on boot. Start by creating the {{Mono|/etc/rc.conf}} system configuration file.
 
{{Code|<nowiki>cat >> /etc/rc.conf << _EOF_</nowiki>}}
{{Output|<nowiki>hostname="VM.privatebox"
ifconfig_em0="DHCP"
geli_swap_flags="-e aes -l 256 -s 4096 -d"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
sshd_enable="YES"
tmpmfs="YES"
tmpsize="512m"
tmpmfs_flags="-m 0 -o async,noatime -S -p 1777"
zfs_enable="YES"
_EOF_</nowiki>}}
 
Next we need to create the {{Mono|/boot/loader.conf}} boot configuration file. Let us start by setting some defaults and then telling it which kernel modules you need loaded on boot. Finally let's tell the loader which zpool to boot from.
 
{{Code|<nowiki>cat >> /boot/loader.conf << _EOF_</nowiki>}}
{{Output|<nowiki>autoboot_delay="5"
loader_logo="beastie"
ahci_load="YES"
geom_eli_load="YES"
geom_mirror_load="YES"
geom_label_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot/root"
_EOF_</nowiki>}}
 
Now that we have our default configuration lets set a root password, fix the local time zone, create a mail aliases database, and exit out of the chroot environment.
 
{{Code|passwd}}
 
{{Code|tzsetup}}
 
{{Code|cd /etc/mail ; make aliases}}
 
{{Code|exit}}
 
==Finishing the Installation==
Now that FreeBSD is installed lets setup the partitions that we designated as SWAP. For this we are going to {{ManPage|man=gmirror|section=8}} the partitions together, add them into fstab, and finally set some variables.
 
{{Code|gmirror label -b prefer swap gpt/swap0 gpt/swap1}}
 
{{Code|<nowiki>cat >> /mnt/etc/fstab << _EOF_</nowiki>>}}
{{Output|<nowiki># [Device]  [Mountpoint] [FStype] [Options]  [Dump]  [Pass#]
/dev/mirror/swap.eli  none  swap  sw  0  0
_EOF_</nowiki>}}
 
Finally we need to setup the new mount points for each zpool and set the bootable volume. Afterwards go ahead and {{ManPage|man=reboot|section=8}}.
 
{{Code|zfs set mountpoint{{=}}/ zroot}}
 
{{Code|zpool set bootfs{{=}}zroot/root zroot}}
 
{{Code|reboot now}}
 
At this point your FreeBSD system is fully installed with a very minimal base system. Exit out of the shell prompt and BSDInstaller, remove the installation media and reboot the machine.


<nowiki>
[shell]OK[/shell][cmd]load ahci[/cmd]
[output]
[o]/boot/kernel/ahci.ko size 0x10450 at 0x14c5000[/o]
[/output]
[shell]OK[/shell][cmd]boot[/cmd][br]
[p]The computer will now go through its boot up process and present you with [man=sysinstall section=8] and the country selection. Choose your country and press enter. If you choose [mono]United States[/mono] the standard American keyboard map will be used. Otherwise you will be presented with the keyboard menu. Select the correct keymap if needed and press enter.[/p]
[img=/img/blog/bsdinstall-2.png]Selecting Country Menu[/img]
[p]At last you will be at the main menu. From here navigate to the Fixit menu and choose either [mono]CDROM/DVD[/mono] or [mono]USB[/mono] dependent on the installation media you used.[/p]
[img=/img/blog/bsdinstall-3.png]FreeBSD Boot Loader Menu[/img]
[header]Prepare the Working Shell Environment[/header]
[p]Now facing the Fixit prompt we can build our working shell environment. Connect to the network via dhcp with [man=dhclient section=8] (for network interface name use [man=ifconfig section=8]). Then create a basic shell startup script and properly link the shell.[/p][br]
[shell]Fixit#[/shell][cmd]mkdir /var/db/[/cmd]
[shell]Fixit#[/shell][cmd]dhclient em0[/cmd][br]
[shell]Fixit#[/shell][cmd]mkdir /root/[/cmd]
[shell]Fixit#[/shell][cmd]export PATH=/mnt2/usr/bin:/mnt2/sbin:$PATH[/cmd]
[shell]Fixit#[/shell][cmd]echo 'setenv PATH' $PATH > /root/.cshrc[/cmd]
[shell]Fixit#[/shell][cmd]cat >> /root/.cshrc << _EOF_[/cmd]
[output]
[o]setenv LANG "en_US.UTF-8"[/o][br]
[o]setenv GEOM_LIBRARY_PATH /mnt2/lib/geom:/lib/geom[/o][br]
[o]set autolist[/o][br]
[o]set color[/o][br]
[o]set filec[/o][br]
[o]set nobeep[/o][br]
[o]set prompt="# "[/o][br]
[o]_EOF_[/o]
[/output]
[shell]Fixit#[/shell][cmd]ln -s /dist/bin/tcsh /bin/tcsh[/cmd]
[shell]Fixit#[/shell][cmd]ln -s /dist/bin/tcsh /bin/csh[/cmd][br]
[p]Create our ssh server keys via [man=ssh-keygen section=1] and then load [man=sshd section=8] so that all this can be done remotely after the initial setup.[/p][br]
[shell]Fixit#[/shell][cmd]mkdir /usr/bin/[/cmd]
[shell]Fixit#[/shell][cmd]mkdir /etc/ssh/[/cmd]
[shell]Fixit#[/shell][cmd]cp /dist/etc/ssh/sshd_config /etc/ssh/[/cmd]
[shell]Fixit#[/shell][cmd]echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config[/cmd]
[shell]Fixit#[/shell][cmd]ssh-keygen -f /etc/ssh/ssh_host_key[wrap]-t rsa1 -b 4096 -N ''[/cmd]
[shell]Fixit#[/shell][cmd]ssh-keygen -f /etc/ssh/ssh_host_dsa_key[wrap]-t dsa -N ''[/cmd]
[shell]Fixit#[/shell][cmd]ssh-keygen -f /etc/ssh/ssh_host_rsa_key[wrap]-t rsa -N ''[/cmd]
[shell]Fixit#[/shell][cmd]ln -s /dist/usr/bin/ssh /usr/bin/ssh[/cmd]
[shell]Fixit#[/shell][cmd]/mnt2/usr/sbin/sshd[/cmd]
[header]Remote Installation[/header]
[p]At this point you should have a basic working environment setup and an ssh daemon running on the machine from the live filesystem. Connect to the box remotely using [man=ssh section=1] and the IP address that was assigned to it by dhclient.[foot]3[/foot][/p][br]
[shell]#[/shell][cmd]ssh -l root 192.168.1.200[/cmd][br]
[p]After you are connected to the box make some basic symlinks and load the required kernel modules via [man=kldload section=8].[/p][br]
[shell]#[/shell][cmd]ln -s /dist/boot/kernel /boot/kernel[/cmd]
[shell]#[/shell][cmd]ln -s /dist/lib /lib[/cmd]
[shell]#[/shell][cmd]kldload geom_mirror[/cmd]
[shell]#[/shell][cmd]kldload opensolaris[/cmd]
[shell]#[/shell][cmd]kldload zfs[/cmd]
[header]Partition the Disks[/header]
[p]Moving on to the disk setup. First locate all of your disks, we can use [man=dmesg section=8] and [man=grep section=1] for this. The system I am using for this guide has two disks, your results may vary.[/p][br]
[shell]#[/shell][cmd]dmesg | grep -e "^ad" -e "^da" -e "^hd"[/cmd]
[output]
[o]ada0 at ahcich0 bus 0 scbus0 target 0 lun 0[/o][br]
[o]ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device[/o][br]
[o]ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)[/o][br]
[o]ada0: Command Queueing enabled[/o][br]
[o]ada0: 512000MB (1048576000 512 byte sectors: 16H 63S/T 16383C)[/o][br]
[o]ada1 at ahcich1 bus 0 scbus1 target 0 lun 0[/o][br]
[o]ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device[/o][br]
[o]ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)[/o][br]
[o]ada1: Command Queueing enabled[/o][br]
[o]ada1: 512000MB (1048576000 512 byte sectors: 16H 63S/T 16383C)[/o]
[/output][br]
[p]For partition table setup we will be using GUID Partition Table (GPT) via [man=gpart section=8]. Let's first pull up a list of the current partitions.[/p][br]
[shell]#[/shell][cmd]gpart show[/cmd][br]
[p]If there are existing partitions on any disk remove them with the delete command, then destroy the disks partitioning scheme.[/p][br]
[shell]#[/shell][cmd]gpart delete -i 1 ada0[/cmd]
[shell]#[/shell][cmd]gpart destroy ada0[/cmd][br]
[p]Create a new partitioning scheme on each disk.[/p][br]
[shell]#[/shell][cmd]gpart create -s gpt ada0[/cmd]
[shell]#[/shell][cmd]gpart create -s gpt ada1[/cmd][br]
[p]Now we need to create all of the disk partitions, we will be creating four partitions on each disk. The first partitions will contain the boot loader, the second partitions are the swap partitions and finally the third partitions are for the zpool (ZFS Pool). Adjust the size of the zpool partition to fit your disk size needs.[/p][br]
[shell]#[/shell][cmd]gpart add -s 64K -t freebsd-boot ada0[/cmd]
[shell]#[/shell][cmd]gpart add -s 4G -t freebsd-swap -l swap0 ada0[/cmd]
[shell]#[/shell][cmd]gpart add -s 495G -t freebsd-zfs -l zroot0 ada0[/cmd][br]
[p]Now that we have the base partition setup, replicate this onto the other disk.[/p][br]
[shell]#[/shell][cmd]gpart add -s 64K -t freebsd-boot ada1[/cmd]
[shell]#[/shell][cmd]gpart add -s 4G -t freebsd-swap -l swap1 ada1[/cmd]
[shell]#[/shell][cmd]gpart add -s 495G -t freebsd-zfs -l zroot1 ada1[/cmd][br]
[p]Install the protected MBR boot code and gptzfsboot loader onto each of the disks. The gptzfsboot loader will give you the ability to load zfsloader from a zpool.[/p][br]
[shell]#[/shell][cmd]gpart bootcode -p /mnt2/boot/gptzfsboot -i 1 [wrap]-b /mnt2/boot/pmbr ada0[/cmd]
[shell]#[/shell][cmd]gpart bootcode -p /mnt2/boot/gptzfsboot -i 1 [wrap]-b /mnt2/boot/pmbr ada1[/cmd][br]
[p]Verify the partition table to make sure all of your disks match.[/p][br]
[shell]#[/shell][cmd]gpart show[/cmd]
[output]
[o]=>        34  1048575933  ada0  GPT  (500G)[/o][br]
[o]          34        128    1  freebsd-boot  (64K)[/o][br]
[o]        162    8388608    2  freebsd-swap  (4.0G)[/o][br]
[o]    8388770  1038090240    3  freebsd-zfs  (495G)[/o][br]
[o]  1046479010    2096957        - free -  (1.0G)[/o][br][br]
[o]=>        34  1048575933  ada1  GPT  (500G)[/o][br]
[o]          34        128    1  freebsd-boot  (64K)[/o][br]
[o]        162    8388608    2  freebsd-swap  (4.0G)[/o][br]
[o]    8388770  1038090240    3  freebsd-zfs  (495G)[/o][br]
[o]  1046479010    2096957        - free -  (1.0G)[/o][br][br]
[/output][br]
[p]Take note that I did not use the entire disk, the reason behind this is that when using RAID and replacing problematic disks more than likely your new disk will have a slightly different disk geometry and as such might limit your ability to use it as a replacement disk if you can not fit all the necessary partitions on the disk.[/p]
[header]ZFS Filesystems[/header]
[p]Moving onto the ZFS volume setup, make the temporary directory for the zpool cache. Then create the main [man=zpool section=8] and the root volume, then mount the root volume on to [mono]/mnt[/mono].[/p][br]
[shell]#[/shell][cmd]mkdir /boot/zfs/[/cmd]
[shell]#[/shell][cmd]zpool create -O checksum=fletcher4 -O atime=off[wrap]-O canmount=off -O mountpoint=/mnt [br]-O setuid=off zroot mirror gpt/zroot0 gpt/zroot1[/cmd]
[shell]#[/shell][cmd]zfs create -o mountpoint=legacy -o setuid=on[wrap]zroot/root[/cmd]
[shell]#[/shell][cmd]mount -t zfs zroot/root /mnt[/cmd][br]
[p]Create the [mono]/usr[/mono] container volume and then all the sub-volumes.[/p][br]
[shell]#[/shell][cmd]zfs create -o canmount=off zroot/usr[/cmd]
[shell]#[/shell][cmd]zfs create zroot/usr/home[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=lzjb -o exec=off[wrap]zroot/usr/obj[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=gzip zroot/usr/ports[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=off -o exec=off[wrap]zroot/usr/ports/distfiles[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=off -o exec=off[wrap]zroot/usr/ports/packages[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=gzip -o exec=off[wrap]zroot/usr/src[/cmd]
[shell]#[/shell][cmd]cd /mnt ; ln -s /usr/home home[/cmd][br]
[p]Then the [mono]/var[/mono] container and subsequent sub-volumes.[/p][br]
[shell]#[/shell][cmd]zfs create -o exec=off -o canmount=off zroot/var[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=lzjb zroot/var/audit[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=lzjb zroot/var/crash[/cmd]
[shell]#[/shell][cmd]zfs create zroot/var/db[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=lzjb -o exec=on[wrap]zroot/var/db/pkg[/cmd]
[shell]#[/shell][cmd]zfs create zroot/var/empty[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=gzip zroot/var/log[/cmd]
[shell]#[/shell][cmd]zfs create -o compress=gzip zroot/var/mail[/cmd]
[shell]#[/shell][cmd]zfs create zroot/var/run[/cmd]
[header]FreeBSD Installation[/header]
[p]Now that the ZFS filesystem hierarchy is setup we can install the FreeBSD base system onto the zpool. If you are installing a different release replace [mono]9.*[/mono] with the proper release number.[/p][br]
[shell]#[/shell][cmd]setenv DESTDIR /mnt[/cmd]
[shell]#[/shell][cmd]cd /dist/9.*[/cmd][br]
[p]Install the base, manpages, source, and generic kernel. Then copy the generic kernel over to default. For this part ignore the [mono]tar: Failed to set default locale[/mono] warnings.[/p][br]
[shell]#[/shell][cmd]cd base ; ./install.sh[/cmd]
[shell]#[/shell][cmd]cd ../manpages ; ./install.sh[/cmd]
[shell]#[/shell][cmd]cd ../kernels ; ./install.sh generic[/cmd]
[shell]#[/shell][cmd]cd /mnt/boot ; cp -Rlp GENERIC/*[wrap]/mnt/boot/kernel/[/cmd][br]
[p]Switch the [mono]base/var/empty[/mono] zpool volume to readonly and then chroot into your newly installed system.[/p][br]
[shell]#[/shell][cmd]zfs set readonly=on zroot/var/empty[/cmd]
[shell]#[/shell][cmd]chroot /mnt[/cmd]
[header]FreeBSD Configuration[/header]
[p]Moving onto configuration, setup the network and necessary startup files so the zpool will be mounted automatically on boot. Start by creating the [mono]/etc/rc.conf[/mono] system configuration file.[/p][br]
[shell]#[/shell][cmd]cat >> /etc/rc.conf << _EOF_[/cmd]
[output]
[o]hostname="VM.privatebox"[/o][br]
[o]ifconfig_em0="DHCP"[/o][br]
[o]geli_swap_flags="-e aes -l 256 -s 4096 -d"[/o][br]
[o]ntpd_enable="YES"[/o][br]
[o]ntpd_sync_on_start="YES"[/o][br]
[o]sshd_enable="YES"[/o][br]
[o]tmpmfs="YES"[/o][br]
[o]tmpsize="512m"[/o][br]
[o]tmpmfs_flags="-m 0 -o async,noatime -S -p 1777"[/o][br]
[o]zfs_enable="YES"[/o][br]
[o]_EOF_[/o]
[/output][br]
[p]Next we need to create the [mono]/boot/loader.conf[/mono] boot configuration file. Let us start by setting some defaults and then telling it which kernel modules you need loaded on boot. Finally lets tell the loader which zpool to boot from.[/p][br]
[shell]#[/shell][cmd]cat >> /boot/loader.conf << _EOF_[/cmd]
[output]
[o]autoboot_delay="5"[/o][br]
[o]loader_logo="beastie"[/o][br]
[o]ahci_load="YES"[/o][br]
[o]geom_eli_load="YES"[/o][br]
[o]geom_mirror_load="YES"[/o][br]
[o]geom_label_load="YES"[/o][br]
[o]zfs_load="YES"[/o][br]
[o]vfs.root.mountfrom="zfs:zroot/root"[/o][br]
[o]_EOF_[/o]
[/output][br]
[p]Now that we have our default configuration lets set a root password, fix the local timezone and create a mail aliases database.[/p][br]
[shell]#[/shell][cmd]passwd[/cmd]
[shell]#[/shell][cmd]tzsetup[/cmd]
[shell]#[/shell][cmd]cd /etc/mail ; make aliases[/cmd][br]
[p]Exit from the chroot environment and copy over the zpool cache files onto the base zpool.[/p][br]
[shell]#[/shell][cmd]exit[/cmd]
[shell]#[/shell][cmd]cp /boot/zfs/zpool.cache /mnt/boot/zfs/[/cmd]
[header]Finish the Installation[/header]
[p]Now that FreeBSD is installed lets setup the partitions that we designated as SWAP. For this we are going to [man=gmirror section=8] the partitions together, add them into fstab, and finally set some variables.[/p][br]
[shell]#[/shell][cmd]gmirror label -b prefer swap gpt/swap0[wrap]gpt/swap1[/cmd]
[shell]#[/shell][cmd]cat >> /mnt/etc/fstab << _EOF_[/cmd]
[output]
[o]# Device      Mountpoint FStype Options Dump Pass#[/o][br]
[o]/dev/mirror/swap.eli none swap  sw      0    0[/o][br]
[o]_EOF_[/o]
[/output]
[shell]#[/shell][cmd]cd ; setenv LD_LIBRARY_PATH /mnt2/lib[/cmd][br]
[p]Finally we need to setup the new mount points for each zpool and set the bootable volume. Afterwards go ahead and [man=reboot section=8].[/p][br]
[shell]#[/shell][cmd]zfs set mountpoint=/ zroot[/cmd]
[shell]#[/shell][cmd]zpool set bootfs=zroot/root zroot[/cmd]
[header]Completing the Installation[/header]
[p]At this point your FreeBSD system is fully installed with a very minimal base system. Exit out of the Fixit prompt and sysinstall, remove the installation media and reboot the machine.[/p][br]
[p]From here you should move onto my FreeBSD Initial Setup guide, which goes through creation of users, basic system updating, and mild system hardening (coming soon!).[/p]
[hr]
[footnote=1]FreeBSD Base r197221, Tue Sep 15 2009[br][url=http://svn.freebsd.org/viewvc/base?view=revision&revision=197221]http://svn.freebsd.org/viewvc/base?view=revision&revision=197221[/url][/footnote]
[br]
[footnote=2]FreeBSD Base r199714, Mon Nov 23 2009[br][url=http://svn.freebsd.org/viewvc/base?view=revision&revision=199714]http://svn.freebsd.org/viewvc/base?view=revision&revision=199714[/url][/footnote]
[br]
[footnote=3]If you are not using a Linux or BSD derived operating system you can alternatively use [url=http://www.chiark.greenend.org.uk/~sgtatham/putty/]PuTTY[/url][/footnote]
</nowiki>
==Notes==
==Notes==
{{Reflist}}
<references/>
<br>
<br>
[[Category:FreeBSD]]
[[Category:FreeBSD]]

Latest revision as of 18:30, 10 July 2017

The following is how to install FreeBSD on ZFS root using GPT. This will offer redundancy and scaling of performance (with additional drives) for you to base your system off of while maintaining efficiency of disk usage. This guide is mainly suited for workstations, home file servers, and media center PCs. It is setup for use with FreeBSD 9+ and was directly tested with 9.0-RELEASE version of FreeBSD which includes the new BSDInstall[1].

Introduction to ZFS

ZFS became part of FreeBSD on 6th April 2007, while it stayed in the experimental phase until the 15th of September 2009[2] when they removed the experimental tag and deemed it ready for production use. Then on the 23th of November 2009[3] zfsloader was merged into -STABLE making ZFS as boot option easier than ever.

ZFS and other newer features that have been recently added into FreeBSD are not available through the generic sysinstall(8) that is included on all of the default installation media. To remedy this, I am going to show you how to install FreeBSD manually from the Fixit livefs command line.

Booting the Installation Media

At this point you should boot from your installation media. For use in creation of this article the following test environment was used.

Icon VirtualBox 4.0.14_OSE r74382
Icon FreeBSD-10.0-HEAD-20120120-JPSNAP-amd64-amd64-release.iso

When you first boot up your chosen media you will see the normal boot menu, feel free to wait out the allotted time or press enter. After the system is done booting up you will be presented with the 'Welcome' window inside the new BSDInstaller.

Configuring the Installation Environment

Select < Shell > from the options and press enter. You should now be presented with the shell. First thing we need to do is switch /tmp from read-only to writable (used later to store the zpool.cache file) and then load the required kernel modules via kldload(8).

umount /dev/md1

Ignore any errors you may or may not see from executing this command, the /tmp directory will still be unmounted. Remount the tmp directory and load the needed kernel modules.

mdmfs -s 512M md1 /tmp

kldload geom_mirror

kldload opensolaris

kldload zfs

Remote Installation Setup (Optional)

This section takes two things for granted; (a) that you have a personal set of SSH keys, and (b) that they are located on another server on the network, one that is already running an SCP capable SSH server. If you do not have one of these requirements feel free to skip this section or pause to set this up on your own. I feel it is easier to install via SSH due to the large amount of actual typing you are required to do, whereas with SSH you can merely copy/paste into the SSH window.

Connect to the network via DHCP with dhclient(8) (for network interface name use ifconfig(8)), then create our SSH host keys via ssh-keygen(1).

dhclient em0

cd /tmp

ssh-keygen -f /tmp/hostkey -t rsa1 -b 4096 -N ''

ssh-keygen -f /tmp/hostkey_dsa -t dsa -N ''

Next we will need to download your personal SSH keys from the network. Replace the username, IP address, and folder/file with the required information. After that is done go ahead and start up the SSH daemon with sshd(8).

scp user@192.168.x.x:~/.ssh/authorized_keys .

/usr/sbin/sshd -o PermitRootLogin=yes -o HostKey=/tmp/hostkey -o HostDSAKey=/tmp/hostkey_dsa -o AuthorizedKeysFile=/tmp/authorized_keys

Connect to the IP address that was assigned to the installation computer using ssh(1) with your personal SSH keys.[4]

ssh -l root 192.168.x.x

Disk Partition Setup

Moving on to the disk setup, first locate all of your disks, we can use dmesg(8) and grep(1) for this. The system I am using for this guide has two disks, your results may vary.

dmesg | grep -e "^ad" -e "^da" -e "^hd"

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 512000MB (1048576000 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 512000MB (1048576000 512 byte sectors: 16H 63S/T 163

For partition table setup we will be using GUID Partition Table (GPT) via gpart(8). Let's first pull up a list of the current partitions.

gpart show

If there are existing partitions on any disk remove them with the delete command, then destroy the disks partitioning scheme.

gpart delete -i 1 ada0

gpart destroy ada0

Create a new partitioning scheme on each disk.

gpart create -s gpt ada0

gpart create -s gpt ada1

Now we need to create all of the disk partitions, we will be creating four partitions on each disk. The first partitions will contain the boot loader, the second partitions are the swap partitions and finally the third partitions are for the zpool (ZFS Pool). Adjust the size of the zpool partition to fit your disk size needs.

gpart add -s 64K -t freebsd-boot ada0

gpart add -s 4G -t freebsd-swap -l swap0 ada0

gpart add -s 495G -t freebsd-zfs -l zroot0 ada0

Now that we have the base partition setup, replicate this onto the other disk.

gpart add -s 64K -t freebsd-boot ada1

gpart add -s 4G -t freebsd-swap -l swap1 ada1

gpart add -s 495G -t freebsd-zfs -l zroot1 ada1

Install the protected MBR boot code and gptzfsboot loader onto each of the disks. The gptzfsboot loader will give you the ability to load zfsloader from a zpool.

gpart bootcode -b /boot/pmbr -i 1 -p /boot/gptzfsboot ada0

gpart bootcode -b /boot/pmbr -i 1 -p /boot/gptzfsboot ada1

Verify the partition table to make sure all of the disks match.

gpart show

=> 34 1048575933 ada0 GPT (500G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 1038090240 3 freebsd-zfs (495G) 1046479010 2096957 - free - (1.0G) => 34 1048575933 ada1 GPT (500G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 1038090240 3 freebsd-zfs (495G) 1046479010 2096957 - free - (1.0G)

Take note that I did not use the entire disk, the reason behind this is that when using RAID and replacing problematic disks more than likely your new disk will have a slightly different disk geometry and as such might limit your ability to use it as a replacement disk if you cannot fit all the necessary partitions on the disk.

ZFS File System

Moving onto the ZFS volume setup, create the main zpool(8) and the root volume, then mount the root volume on to /mnt.

zpool create -O atime=off -O canmount=off -O checksum=fletcher4 -O mountpoint=/mnt -O setuid=off zroot raidz gpt/zroot0 gpt/zroot1

zfs create -o mountpoint=legacy -o setuid=on zroot/root

mount -t zfs zroot/root /mnt

Create the /usr container volume and then all the sub-volumes.

zfs create -o canmount=off zroot/usr

zfs create zroot/usr/home

zfs create -o compress=lzjb -o exec=on zroot/usr/obj

zfs create -o compress=gzip zroot/usr/ports

zfs create -o compress=off -o exec=off zroot/usr/ports/distfiles

zfs create -o compress=off -o exec=off zroot/usr/ports/packages

zfs create -o compress=gzip -o exec=off zroot/usr/src

cd /mnt ; ln -s /usr/home home

Then create the /var container and subsequent sub-volumes.

zfs create -o exec=off -o canmount=off zroot/var

zfs create -o compress=lzjb zroot/var/audit

zfs create -o compress=lzjb zroot/var/crash

zfs create zroot/var/db

zfs create -o compress=lzjb -o exec=on zroot/var/db/pkg

zfs create zroot/var/empty

zfs create -o compress=gzip zroot/var/log

zfs create -o compress=gzip zroot/var/mail

zfs create zroot/var/run

FreeBSD Installation

Now that the ZFS file system hierarchy is setup we can remount the ZFS pool, install the FreeBSD base system, and then copy the zpool.cache. When you import the zpool back into the system you might get errors about not being able to create mountpoints, this is rectified two commands later.

cd

zpool export zroot

zpool import -o cachefile=/tmp/zpool.cache zroot

mount -t zfs zroot/root /mnt

zfs mount -a


sh

cd /usr/freebsd-dist

export DESTDIR=/mnt

for file in base.txz lib32.txz kernel.txz doc.txz;

do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done


cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache

exit

Switch the zroot/var/empty zpool volume to read-only and then chroot into your newly installed system.

zfs set readonly=on zroot/var/empty

chroot /mnt

FreeBSD Configuration

Setup the network and necessary startup files so the zpool will be mounted automatically on boot. Start by creating the /etc/rc.conf system configuration file.

cat >> /etc/rc.conf << _EOF_

hostname="VM.privatebox" ifconfig_em0="DHCP" geli_swap_flags="-e aes -l 256 -s 4096 -d" ntpd_enable="YES" ntpd_sync_on_start="YES" sshd_enable="YES" tmpmfs="YES" tmpsize="512m" tmpmfs_flags="-m 0 -o async,noatime -S -p 1777" zfs_enable="YES" _EOF_

Next we need to create the /boot/loader.conf boot configuration file. Let us start by setting some defaults and then telling it which kernel modules you need loaded on boot. Finally let's tell the loader which zpool to boot from.

cat >> /boot/loader.conf << _EOF_

autoboot_delay="5" loader_logo="beastie" ahci_load="YES" geom_eli_load="YES" geom_mirror_load="YES" geom_label_load="YES" zfs_load="YES" vfs.root.mountfrom="zfs:zroot/root" _EOF_

Now that we have our default configuration lets set a root password, fix the local time zone, create a mail aliases database, and exit out of the chroot environment.

passwd

tzsetup

cd /etc/mail ; make aliases

exit

Finishing the Installation

Now that FreeBSD is installed lets setup the partitions that we designated as SWAP. For this we are going to gmirror(8) the partitions together, add them into fstab, and finally set some variables.

gmirror label -b prefer swap gpt/swap0 gpt/swap1

cat >> /mnt/etc/fstab << _EOF_>

# [Device] [Mountpoint] [FStype] [Options] [Dump] [Pass#] /dev/mirror/swap.eli none swap sw 0 0 _EOF_

Finally we need to setup the new mount points for each zpool and set the bootable volume. Afterwards go ahead and reboot(8).

zfs set mountpoint=/ zroot

zpool set bootfs=zroot/root zroot

reboot now

At this point your FreeBSD system is fully installed with a very minimal base system. Exit out of the shell prompt and BSDInstaller, remove the installation media and reboot the machine.

Notes

  1. ^ "FreeBSD Base r218799, Fri Feb 18 2011". http://svn.freebsd.org/viewvc/base?view=revision&revision=218799. 
  2. ^ "FreeBSD Base r197221, Tue Sep 15 2009". http://svn.freebsd.org/viewvc/base?view=revision&revision=197221. 
  3. ^ "FreeBSD Base r199714, Mon Nov 23 2009". http://svn.freebsd.org/viewvc/base?view=revision&revision=199714. 
  4. ^ "PuTTY". If you are not using a Linux or BSD derived operating system you can alternatively use PuTTY. http://www.chiark.greenend.org.uk/~sgtatham/putty/. 


Retrieved from "https://kyau.net/w/index.php?title=FreeBSD:ZFS_Install&oldid=2449"