/boot on ZFS no longer works with Debian's GRUB after "zpool upgrade"?

Question

Let'see what happen after creating a bpool and upgrading as command zpool status suggest

root@debian:/boot# grub-probe /boot
zfs
root@debian:/boot# zpool upgrade bpool
This system supports ZFS pool feature flags.

Enabled the following features on 'bpool':
  multi_vdev_crash_dump
  large_dnode
  sha512
  skein
  edonr
  userobj_accounting
  encryption
  project_quota
  device_removal
  obsolete_counts
  spacemap_v2
  allocation_classes
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid
  zilsaxattr
  head_errlog
  blake3
  block_cloning
  vdev_zaps_v2
  redaction_list_spill
  raidz_expansion
  fast_dedup
  longname
  large_microzap

root@debian:/boot# grub-probe /boot
grub-probe: error: unknown filesystem

As you can see the fs is not more recognized, and at reboot it won't start (go to grub rescue console, and is not possible to recover it, only live-cd works). I have found a workaround: recreate the bpool with this command on live-cd

zpool create -f -o ashift=12 -d -o feature@async_destroy=enabled -o feature@bookmarks=enabled -o feature@embedded_data=enabled -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled -o feature@large_blocks=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@zpool_checkpoint=enabled -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/boot -R /mnt bpool mirror /dev/disk/by-partlabel/BOOTPOOL1 /dev/disk/by-partlabel/BOOTPOOL2

Anyone know why grub failed to recognize zfs or a better solution instead of recreate bpool?

Answer 1

As Stephane mentioned in the comments, the correct solution to this is to set the compatibility pool option. This is intended to prevent upgrades outside of a known list of compatible features. This option was introduced in 2.1.0, but for older versions leads me to my next point:

I'm going to go a bit further than that and say you should not be upgrading your bpool at all. Generally speaking, new features aren't going to contain useful changes for a small specific-purpose pool used for boot files only. The risk far outweighs the reward here.

More generally, you should be treating zpool upgrade as a potentially dangerous operation, especially if other systems (including rescue disks) may need to import the pool. This means you should zpool checkpoint before the upgrade (and discard the checkpoint after testing). As you've discovered, once you've performed an un-checkpointed upgrade your only option to go back is to recreate the pool.

Also generally the OpenZFS documentation is a good place to refer to for root-on-ZFS installations. You'll note that for bookworm they create the bpool with compatibility=grub2; this was not available in older versions.

Answer 2

Solution found.

sudo zpool status

  pool: bpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.

The more interesting part is this:

Once this is done, the pool may no longer be accessible by software that does not support the features.

So grub2 don't support some features.

Possible solutions:

1. Don't use zfs for boot, is possible to use zfs as root, and the good old raid software with mdadm for boot, or other fs

2. Wait for grub2 to support all features of zfs on boot maybe report which don't work

3. Use a test vm, and try to enable features, one after one until grub-probe fail, so we can see which feature don't work, is possible to enable some features, and some not.