SR-IOV is a First Class Feature

One of my favorite hardware features is called Single-Root Input/Output Virtualization (SR-IOV). In layman's terms, it makes a single physical device appear like multiple similar devices to the operating system.

SR-IOV for Networking

I like to make use of SR-IOV for networking. I have an Intel X710-DA2 Fiber Network Adapter in my FreeBSD server. It has 2 physical fiber ports, which are what SR-IOV refers to as physical functions (PFs). With SR-IOV, I can create several Virtual Functions (VFs) from any PF. So if I configure one physical port to make use of SR-IOV it will look something like like:

Physical Port 1 (PF 1)
|-Virtual Port  1 (VF 1)  <-->  Jail for IRC
|-Virtual Port  2 (VF 2)  <-->  Jail for Web Server
|-Virtual Port  3 (VF 3)  <-->  Jail for File Server
...
|-Virtual Port 16 (VF 16) <-->  Jail for whatever

Like this, the host OS gives the VFs to the jails. Once this happens, the VF is no longer visible in the host with ifconfig(8). The jail fully owns the VF until it's shutdown. This means no software bridges, switches, etc. You just setup the jail as if it had it's own dedicated PCIe network adapter. Super convenient.

SR-IOV Networking in FreeBSD

All you need to do to setup the X710-DA2 is create a file in /etc/iov with some specifics. My card can do 63 VFs, but I set it to 10 because that's all I need (makes looking at ifconfig easier too). Think of specifying 10 as "turning SR-IOV on" and a prerequisite for everything that follows, which are driver and VF specific options, mac addresses, and other SR-IOV specific configuration. Nice an neat. All in one place.

# /etc/iov/ixl1.conf
PF {
        device : "ixl1"
        num_vfs : 10
}

DEFAULT {
        passthrough : false;
        mac-anti-spoof : false;
        allow-set-mac : true;
        allow-promisc : true;
}

VF-0 {
        mac-addr : "aa:88:44:00:02:00";
}

VF-1 {
        mac-addr : "aa:88:44:00:02:01";
}

[...repeat 8 more times...]

One last step to make it effective:

iovctl -C -f /etc/iov/ixl1.conf

The iovctl(8) is part of the base OS. This command sets up the VFs according to the file we just created. What's cool is the ixl(4) and iavf(4) driver manuals have all the possible options well documented. To check that it all works we use ifconfig(8):

ifconfig
...
iavf0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether aa:88:44:00:02:00
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...

SR-IOV in Linux

There is no standard tool for setting up SR-IOV in Linux. There are several ways to do it, and each feels a bit hacky and hidden. I'll highlight what I use, udev.

udev is actually pretty cool and does a lot of stuff. Unfortunately it takes a bit of digging to make it work. Let's start at the finish line:

# /etc/udev/rules.d/30-sriov-numvfs-for-mellanox-connectx4lx.rules
# Set number of virtual functions on NIC
# Upper limit is 63
# PAY ATTENTION TO == VS. =

# To get vendor and device:
# lspci -nn | grep ConnectX

KERNEL=="0000:05:00.1", SUBSYSTEM=="pci", ATTRS{vendor}=="0x15b3", ATTRS{device}=="0x1015", ATTR{sriov_drivers_autoprobe}="1", ATTR{sriov_numvfs}="10"

First, you'll notice I left myself notes because it's not obvious where to get all the info for the last line that does all the work. Also note that == means "act on devices that match this criteria" and "=" means "set this attribute".

lspci will get you the hex PCI address, vendor and device ID.

lspci -nn | grep ConnectX            
05:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
05:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

The two attributes we'll set are from /sys/bus/pci/devices/0000:05:00.1/, which I'm not sure how I would have ever known existed without internet search. Inside are our two targets, sriov_drivers_autoprobe and sriov_numvfs, that udev will set for us at boot time. If we want to set them without a reboot, we could manually write to them, e.g.:

echo '10' > /sys/bus/pci/devices/0000:05:00.1/sriov_numvfs

So that "turns on" the hardware SR-IOV capability, but we need to configure it. This will depend on how you manage networking. For example, in Debian if you use systemd-networkd, you'd do something like this.

#/etc/systemd/network/21-wired-sriov-p1.network
[Match]
Name=enp5s0f1np1                                                                                                                                  

[SR-IOV]                                                                
VirtualFunction=0
Trust=true

[SR-IOV]                            
VirtualFunction=1
Trust=true

[... repeat as needed ...]

With that, we restart the service and the VFs are ready to use.

ip link
...
6: enp5s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether b8:ce:f6:93:c1:97 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust on, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust on, query_rss off
...

FreeBSD SR-IOV is More Apparent

So a one-liner via udev turns on SR-IOV in Linux, but you have to separately configure some of the hardware attributes elsewhere. And neither is done via an "SR-IOV" feature, which means it takes some hunting to figure it out for the first time.

As with other things, you can find your way in FreeBSD:

$ apropos "SR-IOV"
iovctl(8) - PCI SR-IOV configuration utility

That manual will lead you to the config file and the driver pages for the options. When things are apparent and easy to find, system administration doesn't feel like a chore.