Skip to content

FreeBSD's Cohesive Boot Process

When your system powers on, the flow looks something like:

  • POST → EFI → Boot Loader → OS Kernel

There are many more options/details, and some steps might actually be several steps, but for the sake of this article this is good enough. The thing to pay attention to here is the process chain starts the same regardless of OS, but becomes OS unique once you pass the EFI step. So for each OS, it looks a bit more like this:

  • Debian: POST → EFI → GRUB → initramfs → Linux Kernel
  • FreeBSD: POST → EFI → Loader → FreeBSD Kernel

In FreeBSD, the boot process feels a bit more cohesive, i.e., it feels like you're working pieces of the same project. Components of a Linux boot feel like distinct projects that each need attention in their own special way. I'll use two example to highlight what I mean: Serial Over LAN (SOL) and Full Disk Encryption (FDE).

Serial Over LAN (SOL)

SOL is one of my favorite server features. Most server motherboards have management features to allow you to work with the hardware even if it's powered off. SOL is one of these.

With a motherboard with BMC/IPMI you plug a LAN cable into a dedicated management port, which exposes management services to the LAN. Then using software like ipmitool(1) you can do all sorts of things. One of those is setting up a serial connection over a local network. Once connected, it's the same as if you were physically at the computer connected via a serial cable.

This is great for anything that you might do between initiating a reboot and the system coming fully online, e.g., BIOS changes, firmware maintenance, fixing the occasional boot-breaking misconfiguration, and entering full disk encryption passphrases.

SOL Connection
kitty terminal emulator running on macOS and connected to a headless server's /dev/ttyu1 serial device using ipmitool(1) and the server motherboard's SOL capability.

Configuring the SOL Console in FreeBSD

As with most core OS topics, the FreeBSD Handbook explains how to setup the console.

I could stop there. You can literally copy/paste the setup mentioned in the handbook. But let's assume we have no internet connection and see if in-system documentation can lead us to an answer. (By the way, if you install the en-freebsd-doc package you'll have the handbook offline too, but we'll assume you didn't do that.)

It's common FreeBSD knowledge that most OS startup configuration goes in one of two places: /boot/loader.conf and /etc/rc.conf. Since we want console access when the loader is running, the former seems like a reasonable place to start.

  1. We type man loader.conf and read loader.conf(5). In the first paragraph we see we can use loader.conf to "set all variables described in loader(8)."
  2. In loader(8) the BUILTIN ENVIRONMENT VARIABLES section points to loader_simp(8)
  3. Score! loader_simp(8) has all the variables we need to put in /boot/loader.conf
# /boot/loader.conf
boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole"

That was easy. One copy/paste solution from the source, and one we can find in the OS itself. Both official sources of truth. It feels cohesive and intuitive.

Configuring the SOL Console in Linux

In a Linux system, GRUB and Linux boot are two independent things and you need to configure both correctly. It's possible to setup GRUB correctly to work with SOL, but not the Linux kernel. The opposite is true too.

Let's assume the same scenario, we're logged into Debian, but no internet connection. Let's see what we can find.

  1. You start with man grub, but get nothing.
  2. You broaden your search, apropos grub results in 27 options (turns out GRUB is like a mini OS itself). You guess grub-mkconfig is the right direction since we're talking configuration.
  3. You type man grub-mkconfig only to get slapped in the face by GNU's "our way is better" approach:
    The full documentation for grub-mkconfig is maintained as a Texinfo manual. If the info and grub-mkconfig programs are properly installed at your site, the command info grub-mkconfig should give you access to the complete manual.
  4. You type info and find it isn't installed. Isn't that nice? Not only is the documentation missing for the thing that boots your system, but it's your fault! If we kept with the same "can't access the internet" scenario that kept us from the FreeBSD Handbook, we'd likely be stuck at this point. For the sake of finding an answer, let's assume the internet is suddenly accessible. Hooray! It's sure to be easy to get an official answer now.
  5. You don't feel like installing and learning info so you search the web and find grub-mkconfig on the GNU site.
  6. That document points you to another doc called serial, which gives you the syntax for GRUB_SERIAL_COMMAND. Nice, but GRUB is only half the battle. We need to figure out what goes in GRUB_CMDLINE_LINUX_DEFAULT so we can see kernel boot messages after GRUB completes.
  7. You again turn to search and find the Linux Kernel parameters. You search "console" on the page and make your way down to line 745 and find some info.
  8. In that section, you follow a link to a dedicated kernel console documentation page. There you find the input you need for GRUB_CMDLINE_LINUX_DEFAULT

Finally, you assemble your GRUB configuration file:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=tty1 console=ttyS1,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="quiet serial --speed=115200 --unit=1 --word=8 --parity=no --stop=1"

Not the most straightforward path. Definitely not cohesive. To be fair, the Arch Wiki has become a sort of universal Linux handbook. For this specific topic, it'd save us a lot hunting. But without it, the in-system documentation doesn't fill in the gaps much, and even the official documentation online requires a bit of hunting and reliance on search engines.

Full Disk Encryption (FDE)

Full disk encryption is a much simpler example.

In FreeBSD, when you install and choose FDE, geli(8) is used to encrypt the disk. FreeBSD's loader knows about geli and when you boot you are prompted for a passphrase.

Linux can be this simple, but it really depends which distribution you use and how that distribution handles decryption. For example, Debian's udev approach works, but will ignore any kernel parameters you pass for LUKS. Why? That's because between GRUB and Linux is a small, semi-hidden system just for boot called initramfs. It ignores our kernel parameters for LUKS. However, if you setup Arch Linux to use systemd boot, you can pass kernel parameters. It gets confusing fast. There are serveral combinations that result in differences in how /etc/crypttab is read, what goes into an initramfs (if used), etc. If you want proof, attempt to install Gentoo Linux with FDE.

In short, FreeBSD feels more integrated and is covered by the Handbook. With Linux, you have to know the nuanced details of how the distribution maintainers implemented it. This is where running a Debian system and relying on the Arch wiki can get really frustrating. It feels like Debian is broken, but in reality Arch Linux docs weren't written for Debian. So while most of the time the wiki will help, occassionally it will lead you down the wrong path.

Overall Boot Process Cohesion

Zooming back out to the boot process overall, everything is tied together and documented in the same place with FreeBSD. In a subsequent section I'll cover ZFS boot environments, which is yet another cohesive boot feature.

Linux can do all the same things, but you'll likely spend a lot of time figuring out how to do them. One thing I've learned with Linux over the years is take good notes when you install or modify your system for stuff like this. You'll inevitably forget what you did and won't always have an official source to fall back to.