Installing FreeBSD on a mirrored ZFS pool

b0rken.org / freebsd / Installing FreeBSD on a mirrored ZFS pool

Installing FreeBSD on a mirrored ZFS pool

Synopsis:

These notes describe my method of installing FreeBSD on a mirrored ZFS pool. It is based mostly on Scot Hetzel's excellent guide on the FreeBSD Wiki.

My method differs from Scot's in that I employ a ZFS filesystem layout that is based on the default filesystem layout of OpenSolaris, avoiding use of the top-level dataset in the pool as the root filesystem, in accordance with the ZFS Best Practices. It says, "Consider using the top-level dataset as a container for other file systems" which I interpret as meaning "use the top-level dataset only as a container for other filesystems". This helps avoid potential adminstrative difficulties later on, and makes it possible to set up a complete set of filesystems "parallel" with the existing ones, if, for example, you want to create an alternative boot environment. It isn't possible to create a filesystem "parallel" to the top-level dataset.

I also like to keep my user data as seperate from the OS data as possible, so I put the home filesystems into an entirely seperate branch of the ZFS tree. I am thus able to perform recursive ZFS actions on either the OS data or the user data without affecting or including the other.

I have not used GEOM labels for the GPT partitions either. ZFS seems intelligent enough to automatically spot when a device name has changed. When I originally built a FreeBSD box on ZFS, I was unaware of the existence of the AHCI module, so my pool was built on ad4 and ad6 devices. After I enabled the module the disks changed to ada0 and ada1, but ZFS seamlessly continued to recognise their freebsd-zfs partitions as components of the root pool, referencing them by their gptid instead.

Prerequisites:

Process:

  1. Boot from the USB stick or DVD
  2. At the loader menu, press 6 to escape to a loader prompt
  3. Load required modules, then continue booting:
    4. Type '?' for a list of commands, 'help' for more detailed help. OK load ahci /boot/kernel/ahci.ko size 0xfd88 at 0x126b000 OK load zfs /boot/kernel/zfs.ko size 0x19eb18 at 0x127b000 loading required module 'opensolaris' /boot/kernel/opensolaris.ko size 0x3868 at 0x141a000 OK boot
  4. In sysinstall, set your keyboard language and then choose Fixit from the main menu, specifying the appropriate device to run the environment from (DVD or USB)
  5. GPT partition first disk, ada0, and install bootcode:
    6. # gpart create -s gpt ada0 # gpart add -s 64K -t freebsd-boot ada0 ada0p1 added # gpart add -s 4G -t freebsd-swap ada0 ada0p2 added # gpart add -t freebsd-zfs ada0 ada0p3 added # gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ada0 ada0 has bootcode
  6. Repeat the previous steps for second disk, ada1.
  7. Show GPT layout so far:
    8. # gpart show => 34 488397101 ada0 GPT (233G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 480008365 3 freebsd-zfs (229G) => 34 488397101 ada1 GPT (233G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 480008365 3 freebsd-zfs (229G)
  8. Set the single partition defined in the protective MBR as active. Some systems that have a BIOS that checks for the existence of an active MBR partition on the boot drive and fails to boot if it isn't found. Intel motherboards seem to do this:
    9. # fdisk -a -1 /dev/ada0 # fdisk -a -1 /dev/ada1
  9. Create /boot/zfs directory for zpool.cache file:
    10. # mkdir /boot/zfs
  10. Create mirrored zpool from freebsd-zfs partitions on each disk:
    11. # zpool create pool0 mirror ada0p3 ada1p3 # zpool status -v pool0 pool: pool0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors
  11. Create ZFS hierarchy, mounted under /mnt:
    12. # zfs set mountpoint=none pool0 # zfs create pool0/ROOT # zfs create -o mountpount=/mnt pool0/ROOT/freebsd # zfs create pool0/ROOT/freebsd/var # zfs create pool0/ROOT/freebsd/usr # zfs create pool0/ROOT/freebsd/usr/local # zfs create pool0/ROOT/freebsd/usr/ports # zfs create pool0/ROOT/freebsd/usr/ports/distfiles # zfs create pool0/ROOT/freebsd/usr/src # zfs create -o mountpoint=/home pool0/HOME
  12. Extract FreeBSD distribution to ZFS filesystems:
    13. # export DESTDIR=/mnt # cd /dist/8.1-RELEASE # for dir in base catpages dict doc info lib32 manpages; do (cd $dir; ./install.sh); done # cd kernels # ./install.sh GENERIC # cd ../src # ./install.sh all
  13. Copy GENERIC kernel to expected location:
    14. # cd /mnt/boot # cp -Rpv GENERIC/* kernel/
  14. Edit loader.conf and rc.conf to load modules and start ZFS on boot:
    15. # vi /mnt/boot/loader.conf ahci_load="YES" zfs_load="YES" vfs.root.mountfrom="zfs:pool0/ROOT/freebsd" # vi /mnt/etc/rc.conf zfs_enable="YES"
  15. Set up fstab to mount swap and tmpfs* on boot:
    16. # vi /mnt/etc/fstab tmpfs /tmp tmpfs rw,mode=01777 0 0 /dev/ada0p2 none swap sw 0 0 /dev/ada1p2 none swap sw 0 0

    (* Using tmpfs with zfs is apparently not a good idea - see ZFSKnownProblems on the FreeBSD wiki)

  16. Copy zpool.cache to /mnt/boot:
    17. # cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache
  17. Export LD_LIBRARY_PATH:
    18. # export LD_LIBRARY_PATH=/dist/lib
  18. Unmount all ZFS filesystems and then change their mountpoints:
    19. # zfs umount -a # zfs set mountpoint=legacy pool0/ROOT/freebsd # zfs set mountpoint=/usr pool0/ROOT/freebsd/usr # zfs set mountpoint=/var pool0/ROOT/freebsd/var # zfs set mountpoint=/home pool0/home
  19. Set bootfs property on pool:
    20. # zpool set bootfs=pool0/ROOT/freebsd pool0
  20. Exit Fixit environment and reboot

    21. The system should now boot into the new ZFS-based FreeBSD installation. At this stage, it is very minimally configured. The following still needs to be done:

    • Set root's password
    • Configure hostname and networking
    • Configure timezone
    • Configure services to start on boot; sshd, ntpd, etc

Last modified: 2011-07-14 09:54