hermes-skills/ipmi-bmc-management/SKILL.md

---
name: ipmi-bmc-management
description: Use when working with IPMI/BMC out-of-band server mgmt.
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [ipmi, bmc, redfish, supermicro, out-of-band, hardware, firmware]
    related_skills: [proxmox-lxc-deployment, truenas]
---

# IPMI / BMC Out-of-Band Management

Manage servers out-of-band via their BMC (Baseboard Management Controller).
Covers read-only inventory, sensor/thermal readout, firmware version checks,
and the modern Redfish REST API. Credentials live in fact_store, never in this
skill.

## Install

```bash
sudo apt-get install -y ipmitool        # CLI (also: freeipmi-tools)
```

## ipmitool — read-only inventory

```bash
ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> -C 3 <cmd>
```

Useful read-only commands:
- `mc info` — BMC identity (manufacturer, product ID, firmware rev, IPMI version)
- `fru print` — board serial, mfg date, product info
- `sensor list` / `sdr list` — thermals, voltages, fans, GPU temps
- `chassis status` — power state, fault flags, power-restore policy
- `power status` — on/off
- `lan print` — BMC network config (IP, MAC, gateway, cipher suites)
- `user list` — configured BMC accounts
- `sel info` / `sel list` — system event log (check for full/overflow)

## CRITICAL: Supermicro cipher-suite quirk

Supermicro BMCs (H12SSL-i and similar) FAIL the default RMCP+ cipher suite
(17 / SHA256) with:

```
Error in open session response message : invalid role
Error: Unable to establish IPMI v2 / RMCP+ session
```

**Fix: force cipher suite 3** with `-C 3`. This is the single most common
failure when scripting Supermicro IPMI. `-C 17` and `-L ADMINISTRATOR` do NOT
help — only `-C 3` works. IPMI 1.5 (`-I lan`) also works but is insecure.

## Redfish — the modern API (preferred for firmware inventory)

Supermicro BMCs expose a full Redfish REST API. Use it for firmware version
checks — it's cleaner than raw OEM IPMI commands (which often return
"Request data length invalid").

```bash
# Root (confirms Redfish + vendor)
curl -sk -u 'USER:PASS' https://<bmc-ip>/redfish/v1/

# Firmware inventory — BMC, BIOS, CPLD versions in one shot
curl -sk -u 'USER:PASS' https://<bmc-ip>/redfish/v1/UpdateService/FirmwareInventory

# Per-component version
curl -sk -u 'USER:PASS' https://<bmc-ip>/redfish/v1/UpdateService/FirmwareInventory/BMC
curl -sk -u 'USER:PASS' https://<bmc-ip>/redfish/v1/UpdateService/FirmwareInventory/BIOS

# System info (BiosVersion, Model, ProcessorSummary)
curl -sk -u 'USER:PASS' https://<bmc-ip>/redfish/v1/Systems/1
```

FirmwareInventory members are named `BMC`, `BIOS`, `Motherboard_CPLD_1`,
`GPU<n>` etc. Each returns `Version`, `Updateable`, `ReleaseDate`. GPUs show
`Version: None` (not firmware-managed via BMC).

## Firmware currency check

1. Read current versions via Redfish FirmwareInventory (BMC + BIOS + CPLD).
2. Cross-check against Supermicro download center:
   `https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-<board>/BIOS`
   (search result snippets list "BIOS Revision: X.Y" and "BMC Firmware Revision: X.Y.Z").
3. Report the gap. Note: Supermicro BMC downgrades are NOT supported after
   security updates — upgrades are one-way.

## Firmware UPDATE (flash) — method decision

**The H12/H12SS bundle ships SAA (SuperServer Automation Assistant) UEFI, NOT
SUM.** SUM is the older in-band updater; the H12 firmware zip contains
`SAA.efi` + `flash.nsh`. Two hard rules from Supermicro's own package readme:
1. **"Using AFU tool will end up with the BIOS corruption. It should never be
   used!"** — the AMI AFU flasher is forbidden on H12.
2. **BIOS flash on H12SS-and-newer MUST go through the BMC** — DOS/EFI
   standalone BIOS flashing is no longer supported. The BMC web UI uploads the
   .bin to the BMC flashdisk and the BMC does the flash.

| Method | License? | Notes |
|---|---|---|
| **BMC web UI → Maintenance → Firmware Management** | No | The supported path for BOTH BMC and BIOS on H12. Upload the .bin; BMC does the flash. ~2.5 min BMC, a few min BIOS. |
| SAA UEFI shell (`flash.nsh <rom> <user> <pw>`) | No | In-band alternative; requires booting host into EFI shell. |
| Redfish SimpleUpdate | YES (DCMS) | Returns `SMC.1.0.OemLicenseNotPassed` — "Not licensed... DCMS needed" |
| Redfish OEM SmcUpdateService.Install | No | Available (Targets + InstallOptions) but web UI is the documented path |
| ipmitool hpm upgrade | No | Only for .hpm images; Supermicro ships .bin |

**Probe the license block before assuming Redfish works:**
`curl -sk -u 'U:P' https://<bmc>/redfish/v1/UpdateService/SimpleUpdateActionInfo`
— if it returns `OemLicenseNotPassed`, SimpleUpdate is out.

**Upgrade order:** BMC first (closes BMC CVEs, one-way), then BIOS (closes
CPU microcode/AGESA CVEs). BIOS flash requires host OFF. Apply BMC and BIOS in
one session — mismatched BIOS/BMC version pairs can hang at POST code FF.

**Recovery safety net:** Supermicro BMCs keep quad-image redundancy —
FirmwareInventory lists `BMC/Backup_BMC/Golden_BMC/Staging_BMC` (same for
BIOS). A failed flash can boot from Backup/Golden via the web UI "Recover"
option or Redfish OEM Install with `InstallOptions=["Recover"]`.

**Post-flash checks:** re-test the cipher suite (newer BMC firmware may switch
from cipher 3 to 17/SHA256), re-verify login, and re-read FirmwareInventory to
confirm the new version. A DHCP BMC may also change IP after reset — re-find by
MAC (fixed) via ARP scan if the IP moves.

## Security advisories to check

- Supermicro Security Center: https://www.supermicro.com/en/support/security_center
- Supermicro BMC advisories (e.g. July 2026 CVE-2026-3821, CVSS 8.8, SMASH
  arbitrary code execution — fix is a BMC firmware update).
- AMD bulletins (e.g. AMD-SB-7054 / CVE-2025-54502, CVSS 7.1, affects EPYC 7002
  "Rome" — fix is RomePI 1.0.0.P AGESA via BIOS update). Cross-check the board's
  CPU generation against the bulletin's affected list.

## SEL (system event log) health

`sel info` shows `Percent Used` and `Overflow`. If 100% full with
`Overflow: true`, new events are being DROPPED. Clearing is a WRITE op —
confirm with the operator before `sel clear`. Supermicro OEM events often
decode as `Unknown #0xff` (raw vendor events, not standard IPMI).

## Security notes

- Legacy IPMI (RMCP+ cipher 3) is weak; prefer Redfish over HTTPS where possible.
- Supermicro publishes BMC security advisories (e.g. July 2026, CVE-2026-3821,
  up to 8.8 CVSS). Old BMC firmware predates these — flag for update.
- `lan print` shows `Bad Password Threshold` (default 3) and lockout interval —
  repeated bad attempts lock the account. Don't brute-force.

## Pitfalls

- **Cipher 17 fails on Supermicro** — always `-C 3` (see above).
- **Raw OEM commands** (`raw 0x30 0x90 ...`) for BIOS version return
  "Request data length invalid" on many Supermicro boards — use Redfish instead.
- **BMC up ≠ host up.** The BMC can report "System Power: on" while the host
  OS is unreachable (no route to host / ARP INCOMPLETE). Verify host reachability
  separately (SSH port probe) before assuming the box is down.
- **Credentials in fact_store, not here.** Look up the BMC IP/user/pass in
  fact_store before connecting; never hardcode passwords in a skill.
- **Supermicro download-center PDFs are bot-blocked (403).** Release-notes PDFs
  and the firmware download page return 403 to curl/SearXNG/browser. BUT the
  firmware itself is freely mirrored and NOT blocked — download the actual
  .bin files from `https://ftp.abacus.cz/support/FW/MB/SUPERMICRO/BIOS/<board>/`
  (or `https://sm.t3x.net/`), then verify SHA256. Thomas-Krenn also mirrors
  tested-stable bundles at `https://www.thomas-krenn.com/en/download?product=<id>`
  (their `/redx/tools/mb_download.php/...` links are curl-able). Only the
  Supermicro.com release-notes PDFs stay behind the wall — use Thomas-Krenn wiki
  changelogs or search-result snippets for those.

## References

- `references/h12ssl-i-firmware-update.md` — H12SSL-i concrete firmware data,
  security drivers, license-block evidence, known issues, changelog sources.