crunchgen: fix NULL-deref bug introduced in r364647

While porting over the local changes from CheriBSD for upstreaming, I
accidentally committed a broken version of find_entry_point(): we have to
re

crunchgen: fix NULL-deref bug introduced in r364647

While porting over the local changes from CheriBSD for upstreaming, I
accidentally committed a broken version of find_entry_point(): we have to
return NULL if the value is not found instead of a value with
ep->name == NULL, since the checks in main were changed to check ep instead
of ep->name for NULL.

This only matters if the crunched tool cannot be found using normal lookup
and one of the fallback paths is used, so it's unlikely to be triggered
in rescue. However, I noticed that one of our CheriBSD test scripts was
failing to run commands under `su` on minimal disk images where all
binaries are hardlinks to a `cheribsdbox` tool generated with crunchgen.

This also updates the bootstrapping check in Makefile.inc1 to bootstrap
crunchgen up to the next version bump.

Reviewed By: kevans
Differential Revision: https://reviews.freebsd.org/D27474

show more ...

Add lists for customizing legacy and bootstrap-tools.

Reviewed by: arichardson
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D27200

[POWERPC64LE,POWEPCSPE] set default kernel config for powerpc64le and powerpcspe variants

Default KERNCONF for powerpc64le should be GENERIC64, and powerpcspe should
select MPC85XXSPE

Reviewed by:

[POWERPC64LE,POWEPCSPE] set default kernel config for powerpc64le and powerpcspe variants

Default KERNCONF for powerpc64le should be GENERIC64, and powerpcspe should
select MPC85XXSPE

Reviewed by: bdragon,emaste
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D27257

show more ...

Makefile.inc1: remove no-longer required variable

This variable is unsed since r364760 but I forgot to delete it in that commit.

Reported By: bdrewery

fix the docs, this was always wrong... In some cases, DISTDIR is set
automatically by tools via /etc/make.conf, so remind people (me) where
to find where it's set..

It would be nice for someone to

fix the docs, this was always wrong... In some cases, DISTDIR is set
automatically by tools via /etc/make.conf, so remind people (me) where
to find where it's set..

It would be nice for someone to document what DISTDIR is better than:
where the file for a distribution gets installed

show more ...

pkgbase: Add incremental packages

This adds a new target update-packages which will create the new packages
compared to the last run.

This is how to use it:
At this point we cut a release
$ make bu

pkgbase: Add incremental packages

This adds a new target update-packages which will create the new packages
compared to the last run.

This is how to use it:
At this point we cut a release
$ make buildworld ...
$ make buildkernel
$ make packages

There is now a PKG_VERSION directory with latest link pointing to it
Distribute the packages to server

$ something something that update the source tree
$ make buildworld ...
$ make buildkernel
$ make update-packages
You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it.
In PKG_VERSION dir only the packages which differs from the latest run are
named PKG_VERSION, otherwise the old packages are there.

The process is :
Build the new packages in the PKG_VERSION directory
Compare the internal data with the PKG_VERSION_FROM version. The comparison is done
by checking the internal hash of the packages.
By default PKG_VERSION_FROM is set to what the latest link points to.
If the old and new version matches, we rm the new package and cp the old one.

Differential Revision: https://reviews.freebsd.org/D25984

show more ...

flua: Add a libjail module

libjail is pretty small, so it makes for a good proof of concept demonstrating
how a system library can be wrapped to create a loadable Lua module for flua.

* Introduce 3

flua: Add a libjail module

libjail is pretty small, so it makes for a good proof of concept demonstrating
how a system library can be wrapped to create a loadable Lua module for flua.

* Introduce 3lua section for man pages
* Add libjail module

Reviewed by: kevans, manpages
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D26080

show more ...

Significantly speed up mkimg_test

It turns out that the majority of the test time for the mkimg tests isn't
mkimg itself but rather the use of jot and hexdump which can be quite slow
on emulated pla

Significantly speed up mkimg_test

It turns out that the majority of the test time for the mkimg tests isn't
mkimg itself but rather the use of jot and hexdump which can be quite slow
on emulated platforms such as QEMU.

On QEMU-RISC-V this reduces the time for `kyua test mkimg_test` from 655
seconds to 200. And for CheriBSD on QEMU-CHERI this saves 4-5 hours (25%
of the time for the entire testsuite!) since jot ends up triggering slow
functions inside the QEMU emulation a lot.

Reviewed By: lwhsu
Differential Revision: https://reviews.freebsd.org/D26796

show more ...

Fix build with -DBOOTSTRAP_ALL_TOOLS

sbin/sysctl can no longer be bootstrapped on FreeBSD 12 after r366465,
so create a symlink to the host tool instead of trying to build it.

Makefile.inc1: sysent: allow subordinate sysent targets to run in parallel

makesyscalls.lua (and indeed makesyscalls.sh) are both safe to be run in
parallel, so let's do it.

This is a trivial diffe

Makefile.inc1: sysent: allow subordinate sysent targets to run in parallel

makesyscalls.lua (and indeed makesyscalls.sh) are both safe to be run in
parallel, so let's do it.

This is a trivial difference because runtime per-target is pretty small, but
I like seeing it run in parallel when my muscle memory types `make -sj4`.

Reviewed by: brooks, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26594

show more ...

[PowerPC64LE] Set up powerpc.powerpc64le architecture

This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend

[PowerPC64LE] Set up powerpc.powerpc64le architecture

This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend learning project for me and kinda snowballed from
there.

(More to follow momentarily.)

Reviewed by: imp (earlier version), emaste
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26399

show more ...

Prefer bootstrapped tools when running certctl.sh

Otherwise we get lots of warnings when building on Linux/macOS during
installworld:
Scanning /local/scratch/alr48/cheri/output/freebsd-x86/usr/share

Prefer bootstrapped tools when running certctl.sh

Otherwise we get lots of warnings when building on Linux/macOS during
installworld:
Scanning /local/scratch/alr48/cheri/output/freebsd-x86/usr/share/certs/blacklisted for certificates...
install: invalid option -- 'U'
Try 'install --help' for more information.
install: invalid option -- 'U'
....

Reviewed By: kevans
Differential Revision: https://reviews.freebsd.org/D26481

show more ...

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, t

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

show more ...

build: provide a default WARNS for all in-tree builds

The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover p

build: provide a default WARNS for all in-tree builds

The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).

Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.

Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.

There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.

Reviewed by: emaste, brooks, ngie (all earlier version)
Reviewed by: emaste, arichardson (depend-cleanup.sh change)
Differential Revision: https://reviews.freebsd.org/D26455

show more ...

Promote the installworld `certctl rehash` to distributeworld

Contrary to my belief, installworld is not sufficient for getting certs
installed into VM images. Promote the rehash to both installworld

Promote the installworld `certctl rehash` to distributeworld

Contrary to my belief, installworld is not sufficient for getting certs
installed into VM images. Promote the rehash to both installworld and
distributeworld (notably: not stageworld) and rehash the base distdir so we
end up with /etc/ssl/certs populated in the base dist archive. A future
commit will remove the rehash from bsdinstall, which doesn't really need to
happen if they're installed into base.txz.

While here, fix a minor typo: s/CERTCLTFLAGS/CERTCTLFLAGS/

MFC after: 1 week

show more ...

installworld: run `certctl rehash` after installation completes

This was originally introduced back in r360833, and subsequently reverted
because it was broken for -DNO_ROOT builds and it may not ha

installworld: run `certctl rehash` after installation completes

This was originally introduced back in r360833, and subsequently reverted
because it was broken for -DNO_ROOT builds and it may not have been the
correct place for it.

While debatably this may still not be 'the correct place,' it's much cleaner
than scattering rehashes all throughout the tree. brooks has fixed the issue
with -DNO_ROOT by properly writing to the METALOG in r361397.

Do note that this is different than what was originally committed; brooks
had revisions in D24932 that made it actually use the revised unprivileged
mode and write to METALOG, along with being a little more friendly to
foreign crossbuilds and just using the certctl in-tree.

With this change, I believe we should now have a populated /etc/ssl/certs in
the VM images.

MFC after: 1 week

show more ...

Makefile.inc1: remove more old stale depend hacks

Current stale dependency hacks are in tools/build/depend-cleanup.sh.
These ones were almost a year old; remove them from Makefile.inc1.

Ensure that the makewhatis symlink is added in the bootstrap-tools stage

We currently set MK_MAN=no in $BSARGS so MK_MAN_UTILS will also be false
which means that the makewhatis symlink will not be

Ensure that the makewhatis symlink is added in the bootstrap-tools stage

We currently set MK_MAN=no in $BSARGS so MK_MAN_UTILS will also be false
which means that the makewhatis symlink will not be created.
This change fixes the build when using both -DBUILD_WITH_STRICT_TMPPATH and
-DBOOTSTRAP_ALL_TOOLS.

Tested by: andrew
Differential Revision: https://reviews.freebsd.org/D16761

show more ...

Add WITH_/WITHOUT_CLEAN option to replace NO_CLEAN

This allows use of the standard src.conf configuration for controlling
whether the tree is cleaned before build or not. The default is still
to cl

Add WITH_/WITHOUT_CLEAN option to replace NO_CLEAN

This allows use of the standard src.conf configuration for controlling
whether the tree is cleaned before build or not. The default is still
to clean.

Setting either NOCLEAN or NO_CLEAN will mention the new src.conf option.
NOCLEAN remains a .warning, while for now NO_CLEAN is .info.

Reviewed by: bdrewery (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D22762

show more ...

Makefile.inc1: comment .endif to ease finding matching .if

libzfs: Also add the crypto dependency to Makefile.inc1

Reported by: kevans
Discussed with: kevans
Sponsored by: iXsystems, Inc.

Use bootstrapped install(1) install of tools/install.sh in world stage

This should be noticeably faster due to fewer processes being forked and
also handles other flags such as -S or writing to META

Use bootstrapped install(1) install of tools/install.sh in world stage

This should be noticeably faster due to fewer processes being forked and
also handles other flags such as -S or writing to METALOG.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26039

show more ...

Fix running the builddtb target on a noexec file system

Obtained from: CheriBSD

Add necessary Makefile.inc1 infrastructure for building on non-FreeBSD

The most awkward bit in this patch is the bootstrapping of m4:
We can't simply use the host version of m4 since that is not com

Add necessary Makefile.inc1 infrastructure for building on non-FreeBSD

The most awkward bit in this patch is the bootstrapping of m4:
We can't simply use the host version of m4 since that is not compatible
with the flags passed by lex (at least on macOS, possibly also on Linux).
Therefore we need to bootstrap m4, but lex needs m4 to build and m4 also
depends on lex (which needs m4 to generate any files). To work around this
cyclic dependency we can build a bootstrap version of m4 (with pre-generated
files) then use that to build the real m4.

This patch also changes the xz/unxz/dd tools to always use the host version
since the version in the source tree cannot easily be bootstrapped on macOS
or Linux.

Reviewed By: brooks, imp (earlier version)
Differential Revision: https://reviews.freebsd.org/D25992

show more ...

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would a

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

show more ...