FreeBSD’s SES-2 Utility
If you have a headless server, you probably don’t physically look at it often. As such, it’s easy to forget the physical layout of things like storage devices. When a disk fails, there’s a bit of panic as you scramble to deal with it before another disk fails. (Side note: there are interesting statistics showing that HDDs from the same batch tend to fail together. I’ve worked through this scenario. If you buy several disks together, have a plan to act fast should one of them fail.)
You don’t want to make any mistakes when resolving it. Probably the easiest mistake to make is pulling the wrong drive and causing other problems. You want to know the physical location of the failed disk, ideally without shutting down the whole system.
Luckily the SES-2 standard exists for enclosure management and FreeBSD has support for this in the base OS.
Using sesutil
Using sesutil(8) will give you output like this.
sesutil showses0: <AHCI SGPIO Enclosure 2.00>; ID: 3061686369656d30 Desc Dev Model Ident Size/Status Slot 00 ada0 WDC WD100EFAX-68LHPN0 JEJXXXXX 10T Slot 01 ada1 WDC WD100EFAX-68LHPN0 JEK9XXXX 10T Slot 02 ada2 WDC WD140EFFX-68VBXN0 9RH5XXXX 14T Slot 03 ada3 WDC WD140EFFX-68VBXN0 X0GEXXXX 14T Slot 04 ada4 WDC WD140EFFX-68VBXN0 Z2JKXXXX 14T Slot 05 ada5 WDC WD140EFFX-68VBXN0 9RK0XXXX 14T Slot 06 ada6 Samsung SSD 870 EVO 2TB S6PNNJ0W413XXXX 2T Slot 07 ada7 Samsung SSD 870 EVO 2TB S6PNNJ0W413XXXX 2T
Notice the slot numbers. Those correspond to physical locations, which is a good thing to know when pulling a drive. The extra info helps to confirm you’re addressing the correct drive.
sesutil will also let you make the light blink on the
drive bay, which adds yet another safeguard to ensure you don’t pull the
wrong drive.
SES-2 in Linux
There is no SES-2 utility installed on Debian by default. You can
install sg3-utils (it’s also available on FreeBSD), but
it’s a bit like bringing an atomic bomb to a fist fight.
You’ll like need to visit the developer’s site to get some hints on how to start. Of note,
Without any options sg_ses reports the names of the supported diagnostic pages. The SAS disk does support some pages but none that belong to the SES standard. On Linux it helps to list all SCSI devices with the lsscsi utility (including showing the generic device names with the ‘-g’ option; ‘modprobe sg’ may be needed if hyphens appear in the last column)
Ok, so we need to install lsscsi because it’s also not installed by default. On my Debian system, I get the following.
lsscsi -gses0: <AHCI SGPIO Enclosure 2.00>; ID: 3061686369656d30 [0:0:0:0] disk ATA WDC WD201KFGX-68 0A83 /dev/sdb /dev/sg0 [0:0:1:0] disk ATA WDC WD201KFGX-68 0A83 /dev/sda /dev/sg1 [0:0:2:0] disk ATA Samsung SSD 870 2B6Q /dev/sdc /dev/sg2 [0:0:3:0] disk ATA WDC WD201KFGX-68 0A83 /dev/sdf /dev/sg3 [0:0:4:0] disk ATA Samsung SSD 870 2B6Q /dev/sde /dev/sg4 [0:0:5:0] disk ATA WDC WD201KFGX-68 0A83 /dev/sdd /dev/sg5 [0:0:6:0] enclosu SMC SC826-P 100b - /dev/sg6 [N:0:4:1] disk Samsung SSD 970 EVO Plus 1TB__1 /dev/nvme0n1 -
You might be thinking those numbers correspond to physical locations, but you’d be wrong. For comparison, here’s the output of a custom script that I wrote to identify drive locations.
report_physical.sh╭─02────────────┬─05────────────┬─08────────────┬─11────────────╮ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ├─01:sdb[LB]────┼─04:sda[LB]────┼─07:sdf[LB]────┼─10:sdd[LB]────┤ │ WDC 20TB 7200 │ WDC 20TB 7200 │ WDC 20TB 7200 │ WDC 20TB 7200 │ │ SATA 3.5 6Gbs │ SATA 3.5 6Gbs │ SATA 3.5 6Gbs │ SATA 3.5 6Gbs │ │ 37C 1.2 yrs │ 37C 1.2 yrs │ 37C 1.1 yrs │ 39C 1.1 yrs │ ├─00────────────┼─03────────────┼─06:sdc[LB]────┼─09:sde[LB]────┤ │ │ │ SAM 2TB SSD │ SAM 2TB SSD │ │ │ │ SATA 3.3 6Gbs │ SATA 3.3 6Gbs │ │ │ │ 34C 0.4 yrs │ 34C 0.4 yrs │ ╰───────────────┴───────────────┴───────────────┴───────────────╯
As we see, slots 1, 4, 6, 7, 9, and 10 are populated.
Now things get a lot more complicated. Running
sg_ses leaves you with hundreds of lines of hard to read
info. I’m sure it makes sense to someone, but it’s pretty intense. You
won’t find a direct answer in the output either. What you find is a
trail of bread crumbs that you’ll need to follow. For example, in the
output below I know Slot 0 is empty because “target port for:” is empty,
whereas Slot 1 shows a SATA device.
Correlating all this will be tedious.
sg_ses -a /dev/bsg/0:0:6:0[...lines omitted...] Slot00 [0,0] Element type: Array device slot [360/4176] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: Not installed OK=0, Reserved device=0, Hot spare=0, Cons check=0 In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0 App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0 Ready to insert=0, RMV=0, Ident=0, Report=0 App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0 Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0 Additional Element Status: Transport protocol: SAS number of phys: 1, not all phys: 0, device slot number: 0 phy index: 0 SAS device type: no SAS device attached initiator port for: target port for: attached SAS address: 0x0 SAS address: 0x0 phy identifier: 0x0 Slot01 [0,1] Element type: Array device slot Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK OK=0, Reserved device=0, Hot spare=0, Cons check=0 In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0 App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0 Ready to insert=0, RMV=0, Ident=0, Report=0 App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0 Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0 Additional Element Status: Transport protocol: SAS number of phys: 1, not all phys: 0, device slot number: 1 phy index: 0 SAS device type: no SAS device attached initiator port for: target port for: SATA_device attached SAS address: 0x5003048020cc31bf SAS address: 0x5003048020cc3181 phy identifier: 0x0 [... hundreds of lines omitted ...]
As an alternative, you might be able to visually parse the output of /dev/disk. For example, if your disks are plugged into a SAS expander:
ls /dev/disk/by-pathlrwxrwxrwx 1 root root 9 Aug 11 21:42 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy10-lun-0 -> ../../sdd lrwxrwxrwx 1 root root 9 Aug 11 21:42 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy1-lun-0 -> ../../sdb lrwxrwxrwx 1 root root 9 Aug 13 18:05 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy4-lun-0 -> ../../sda lrwxrwxrwx 1 root root 9 Aug 11 21:42 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy6-lun-0 -> ../../sdc lrwxrwxrwx 1 root root 9 Aug 11 21:42 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy7-lun-0 -> ../../sdf lrwxrwxrwx 1 root root 9 Aug 11 21:42 pci-0000:01:00.0-sas-exp0x5003048020cc31bf-phy9-lun-0 -> ../../sde
Note that “phyX” bit near the end. That correlates to the physical location. In my experience, whether or not this works will depend on your distribution.
It’s not the easiest approach. And when disaster strikes, it’s not likely that you’re going to be in the mood to learn something complex or parse verbose output.
Simplicity = I’ll Use It
I use sesutil because it’s simple. It does a few things
really well, and in a pinch the manual is concise enough to re-educate
yourself quickly on its use.