boot/zfs: Sync the definition of dsl_dataset_phys with OpenZFS

No functional change intended.

MFC after: 1 week

boot/zfs: Add some fields to dsl_dir_phys_t

Most of the first block of pad bytes are now used for space accounting
purposes. No functional change intended.

MFC after: 1 week
Sponsored by: The Free

boot/zfs: Add some fields to dsl_dir_phys_t

Most of the first block of pad bytes are now used for space accounting
purposes. No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

show more ...

stand/libsa: zfs use standard ZFS_EARLY stuff

Now that the minor issues preventing zfs.c from using CFLAGS_EARLY have
been fixed, use that mechanism like everything else that needs the
OpenZFS spl h

stand/libsa: zfs use standard ZFS_EARLY stuff

Now that the minor issues preventing zfs.c from using CFLAGS_EARLY have
been fixed, use that mechanism like everything else that needs the
OpenZFS spl headers. This simplifies things somewhat. Update comments to
document why zfs.c is still special, though in different ways.

Note: We also use the fact that NEED_SOLARIS_BOOLEAN is only defined in
an environment where the solaris compat boolean stuff will be defined
prior to this point (eg, when we're building zfs.c in libsa), but not in
other environments (like when we're building mkimage and stand-alone
boot loaders that don't use libsa). These latter uses should be changed
to use the same ZFS compile env, but aren't as part of this commit.
This has to be done in the same change as the ZFS_EARLY change to not
break zfs.c building for one commit affecting bisectabiltiy.

Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35894

show more ...

stand: Add blake3 support to boot loader

Add the necessary glue to get blake3 building for the boot loaded as
well as connected to the ZFS system so it is useful.

On some platforms, we create refer

stand: Add blake3 support to boot loader

Add the necessary glue to get blake3 building for the boot loaded as
well as connected to the ZFS system so it is useful.

On some platforms, we create references to blake3_sse2_impl and
blake3_sse41_impl ops structs to utilize SIMD. These aren't present on
x86 (since we dind't ask for them), but are on aarch64 with no
implementation. Since we don't want SIMD in the boot loader, have these
all return 'unsupported' always. This should be fixed upstream to allow
more flexibility in this selection, but for now we use this hack to not
modify the sys/contrib/openzfs with difficult to maintain hacks while
an upstreamable solution is found.

tsoome@ did the implementation bits in sys/cddl/boot, and I did the
Makefile work and the aweful blake3_impl_hack.c.

Co-author: tsoome@freebsd.org
Sponsored by: Netflix
Reviewed by: kevans (earlier version)
Differential Revision: https://reviews.freebsd.org/D35750

show more ...

boot/zfs: Extend zfsimpl.h and make it easier to use

Some makefs(8) patches make use of zfsimpl.h (not zfsimpl.c though) to
provide definitions for various on-disk structures. Most of this diff
sim

boot/zfs: Extend zfsimpl.h and make it easier to use

Some makefs(8) patches make use of zfsimpl.h (not zfsimpl.c though) to
provide definitions for various on-disk structures. Most of this diff
simply adds new definitions that are useful.

Also reduce dependencies of the header:
- remove an unused list_node_t field to drop the sys/list.h dependency
- replace CTASSERT with _Static_assert
And fix the declaration of decode_embedded_bp_compressed().

No functional change intended.

Reviewed by: tsoome
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35278

show more ...

libsa: Make the nvlist implementation more self-contained

Move declarations into a new nvlist.h rather than putting everything in
libzfs.h. This makes this nvlist code easier to reuse elsewhere. I

libsa: Make the nvlist implementation more self-contained

Move declarations into a new nvlist.h rather than putting everything in
libzfs.h. This makes this nvlist code easier to reuse elsewhere. In
particular, the nvlist implementation in sys/contrib/libnv does not
provide XDR encoding, but this is needed when reading from or writing to
ZFS pools.

Also:
- Remove references to boolean_t. It has to be a 32-bit int here, so
just reference the underlying type.
- Add includes needed when compiling the nvlist code outside of stand/.

No functional change intended.

Reviewed by: tsoome
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35255

show more ...

Add zstd support to the boot loader.

Add support to the _STANDALONE environment enough bits of the kernel
that we can compile it. We still have a small zstd_shim.c since there
were 3 items that were

Add zstd support to the boot loader.

Add support to the _STANDALONE environment enough bits of the kernel
that we can compile it. We still have a small zstd_shim.c since there
were 3 items that were a bit hard to nail down and may be cleaned up
in the future. These go hand in hand with a number of commits to
sys/sys in the past weeks, should this need be MFCd.

Discussed with: mmacy (in review and on IRC/Slack)
Reviewed by: freqlabs (on openzfs repo)
Differential Revision: https://reviews.freebsd.org/D26218

show more ...

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, t

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

show more ...

zfs: add an option to the bootloader to rewind the ZFS checkpoint

The checkpoints are another way of keeping the state of ZFS.
During the rewind, the pool has to be exported.
This makes checkpoints

zfs: add an option to the bootloader to rewind the ZFS checkpoint

The checkpoints are another way of keeping the state of ZFS.
During the rewind, the pool has to be exported.
This makes checkpoints unusable when using ZFS as root.
Add the option to rewind the ZFS checkpoint at the boot time.
If checkpoint exists, a new option for rewinding a checkpoint will appear in
the bootloader menu.
We fully support boot environments.
If the rewind option is selected, the boot loader will show a list of
boot environments that existed before the checkpoint.

Reviewed by: tsoome, allanjude, kevans (ok with high-level overview)
Differential Revision: https://reviews.freebsd.org/D24920

show more ...

loader: create single zfs nextboot implementation

We should have nextboot feature implemented in libsa zfs code.
To get there, I have created zfs_nextboot() implementation based on
two sources, our

loader: create single zfs nextboot implementation

We should have nextboot feature implemented in libsa zfs code.
To get there, I have created zfs_nextboot() implementation based on
two sources, our current simple textual string based approach with added
structured boot label PAD structure from OpenZFS.

Secondly, all nvlist details are moved to separate source file and
restructured a bit. This is done to provide base support to add nvlist
add/update feature in followup updates.

And finally, the zfsboot/gptzfsboot disk access functions are swapped to use
libi386 and libsa.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25324

show more ...

loader: rewrite zfs reader zap code to use malloc

First step on removing zfs_alloc.

Reviewed by: delphij
Differential Revision: https://reviews.freebsd.org/D23433

loader: rewrite zfs vdev initialization

In some cases the pool discovery will get stuck in infinite loop while setting
up the vdev children.

To fix, we split the vdev setup into two parts, first we

loader: rewrite zfs vdev initialization

In some cases the pool discovery will get stuck in infinite loop while setting
up the vdev children.

To fix, we split the vdev setup into two parts, first we create vdevs based on
configuration we do get from pool label, then, we process pool config from MOS
and update the pool config if needed.

Testing done: confirm previously hung loader is not hung any more.

MFC after: 1 week

show more ...

loader: factor out label and uberblock load from vdev_probe, add MMP checks

Clean up the label read.

loader: we do not support booting from pool with log device

If pool has log device, stop there and tell about it.

loader: calculate physical vdev psize from asize

Since physical device asize is calculated from psize and the asize is stored
in pool label, we can use asize to set the value of psize, which is used

loader: calculate physical vdev psize from asize

Since physical device asize is calculated from psize and the asize is stored
in pool label, we can use asize to set the value of psize, which is used to
calculate the location of the pool labels.

MFC after: 1 week

show more ...

loader: zio_checksum_verify should check byteswap

We do have both native and byteswap checksum callbacks in place but the
selection is not wired.

MFC after: 1 week

loader: support com.delphix:removing

We should support removing vdev from boot pool. Update loader zfs reader
to support com.delphix:removing.

Reviewed by: allanjude
MFC after: 2 weeks
Differential

loader: support com.delphix:removing

We should support removing vdev from boot pool. Update loader zfs reader
to support com.delphix:removing.

Reviewed by: allanjude
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D18901

show more ...

zfsboot: support newer ZFS versions
declare v3 objset size/layout to fix userboot and possibly other loader issues

- fix for userboot assertion failure in zfs_dev_close in free due to out of bounds

zfsboot: support newer ZFS versions
declare v3 objset size/layout to fix userboot and possibly other loader issues

- fix for userboot assertion failure in zfs_dev_close in free due to out of bounds write
- fix for zfs_alloc / zfs_free mismatch assertion failure when booting GPT on BIOS

show more ...

loader should support large_dnode

The zfsonlinux feature large_dnode is not yet supported by the loader.

Reviewed by: avg, allanjude
Differential Revision: https://reviews.freebsd.org/D12288

loader: update symlink support in zfs reader

As the current zfs file system is providing symlink via system attributes, need
to update the code accordingly.

Note, as the zfsboot code does not free

loader: update symlink support in zfs reader

As the current zfs file system is providing symlink via system attributes, need
to update the code accordingly.

Note, as the zfsboot code does not free the memory at this time, the
object list will put some stress on the boot2 heap, eventually we should
address the issue.

Reviewed by: allanjude, smh
Approved by: allanjude (mentor)
Differential Revision: https://reviews.freebsd.org/D9706

show more ...

Add SHA512, skein, large blocks support for loader zfs.

Updated sha512 from illumos.
Using skein from freebsd crypto tree.
Since loader itself is using 64MB memory for heap, updated zfsboot to
use s

Add SHA512, skein, large blocks support for loader zfs.

Updated sha512 from illumos.
Using skein from freebsd crypto tree.
Since loader itself is using 64MB memory for heap, updated zfsboot to
use same, and this also allows to support zfs large blocks.

Note, adding additional features does increate zfsboot code, therefore
this update does increase zfsboot code to 128k, also I have ported gptldr.S
update to zfsldr.S to support 64k+ code.

With this update, boot1.efi has almost reached the current limit of the size
set for it, so one of the future patches for boot1.efi will need to
increase the limit.

Currently known missing zfs features in boot loader are edonr and gzip support.

Reviewed by: delphij, imp
Approved by: imp (mentor)
Obtained from: sha256.c update and skein_zfs.c stub from illumos.
Differential Revision: https://reviews.freebsd.org/D7418

show more ...

Make boot code and loader check for unsupported ZFS feature flags

OpenZFS uses feature flags instead of a zpool version number to track
features since the split from Oracle. In addition to avoiding

Make boot code and loader check for unsupported ZFS feature flags

OpenZFS uses feature flags instead of a zpool version number to track
features since the split from Oracle. In addition to avoiding confusion
on ZFS vs OpenZFS version numbers, this also allows features to be added
to different operating systems that use OpenZFS in different order.

The previous zfs boot code (gptzfsboot) and loader (zfsloader) blindly
tries to read the pool, and if failed provided only a vague error message.

With this change, both the boot code and loader check the MOS features
list in the ZFS label and compare it against the list of features that
the loader supports. If any unsupported feature is active, the pool is
not considered as a candidate for booting, and a helpful diagnostic
message is printed to the screen. Features that are merely enabled via
zpool upgrade, but not in use, do not block booting from the pool.

Submitted by: Toomas Soome <tsoome@me.com>
Reviewed by: delphij, mav
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D6857

show more ...

MFV r274273:

ZFS large block support.

Please note that booting from datasets that have recordsize greater
than 128KB is not supported (but it's Okay to enable the feature on
the pool). This *may*

MFV r274273:

ZFS large block support.

Please note that booting from datasets that have recordsize greater
than 128KB is not supported (but it's Okay to enable the feature on
the pool). This *may* remain unchanged because of memory constraint.

Limited safety belt is provided for mounted root filesystem but use
caution is advised.

Illumos issue:
5027 zfs large block support

MFC after: 1 month

show more ...

MFV r271510:

Enforce 4K as smallest indirect block size (previously the smallest
indirect block size was 1K but that was never used).

This makes some space estimates more accurate and uses less mem

MFV r271510:

Enforce 4K as smallest indirect block size (previously the smallest
indirect block size was 1K but that was never used).

This makes some space estimates more accurate and uses less memory
for some data structures.

Illumos issue:
5141 zfs minimum indirect block size is 4K

MFC after: 2 weeks

show more ...

MFV r267565:

4757 ZFS embedded-data block pointers ("zero block compression")
4913 zfs release should not be subject to space checks

MFC after: 2 weeks