dsynth - Adjust /boot mounts for chroot

* Fix issue where Intel microcode couldn't be installed within the
chroot due to trying to write into /boot.

* Replace /boot null-mount with a /boot tmpfs

dsynth - Adjust /boot mounts for chroot

* Fix issue where Intel microcode couldn't be installed within the
chroot due to trying to write into /boot.

* Replace /boot null-mount with a /boot tmpfs and mkdir /boot/modules.local.
Leave other /boot directories unpopulated for now (we'll fix that if we
hit problems). i.e. no /boot/kernel or anything else is in the chroot
any more.

* This also allows us to remove the /boot/modules.local tmpfs mount we had
before, improving performance slightly.

Reported-by: tuxillo

show more ...

Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their

Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their dependencies using
configuration data and the "dragonfly/base" dport found in
"/usr/src/dports.base".

This feature is intended to be used by the nrelease build in an upcoming
commit, and may also be used manually.

The /usr/dports tree must exist and contains the dports repo to use as the
basis of the dports build. It will be null-mounted in the chroot and the
"/usr/src/dports.base/dragonfly" directory will be null-mounted under it
to provide the "dragonfly/base" meta-port. The DPORTSBASE make variable
may be used to override the location of the main dports repo.

The build-all and install-all targets use a sophisticated chrooted
environment to do a completely clean dports build with dsynth. It
It will actually "make installworld" plus a clean "make distribution"
inside the chroot environment to create the environment, and the host
system's /etc/resolv.conf will be copied in.

build-all - Augmented when used WITH_DPORTS=YES
install-all - Augmented when used WITH_DPORTS=YES
buildportschroot - Build a fresh chroot environment for dsynth
mountports - Do all null mounts and cp's needed for the env
buildports - Run dsynth build dragonfly/base in the env
installports - Install the built ports on the host system (see note)
umountports - Remove the null mounts

WITH_DPORTS=YES - executes buildportschroot, mountports, buildports,
and umountports when run from build-all.

executes mountports, installports, umountports
when run from install-all.

DPORTSBASE=path - Defaults to /usr/dports, may be used to override
the location where the main dports tree can be
found.

NOTE: Installports may require interactivity as the base system might already
have installed ports. Generally speaking it is best to not revert any
ports you might already have installed, so feel free to tell it not to
install anything.

* Added various make targets and augmented build-all and install-all

* /usr/local/sbin added and the stale /usr/pkg/bin removed from _HOSTPATH.

* Add /usr/src/dports.base infrastructure.

It contains the dsynth.ini and pkg.conf files that will be used for the
local dsynth operations. Some paths are replaced with SED when copied
to the chroot. The dsynth.ini is relative to the chroot.

It also contains the dragonfly/base dport which is the meta-port that
dsynth will be told to build.

All other dports related files and directories are null-mounted from
/usr/dports.

* Add /usr/distfiles.base to mtree/BSD.usr.dist. This directory is also
created by the build if it does not exist since the host system might
not have run a new installworld recently.

This directory will be used during build-all (buildports) to store the
distfiles required to build dragonfly/base, so they only need to be
fetched once.

* Add the "-C ConfigBase" option to dsynth to make running it from the
chrooted environment easier. This overrides the /etc/dsynth directory
and removes /etc/dsynth as a fallback.

* dsynth conditionalizes calling syscap_set() for backwards compatibility
with older host systems that might not have been upgraded to include the
feature yet.

show more ...

kernel - Add per-process capability-based restrictions

* This new system allows userland to set capability restrictions which
turns off numerous kernel features and root accesses. These restricti

kernel - Add per-process capability-based restrictions

* This new system allows userland to set capability restrictions which
turns off numerous kernel features and root accesses. These restrictions
are inherited by sub-processes recursively. Once set, restrictions cannot
be removed.

Basic restrictions that mimic an unadorned jail can be enabled without
creating a jail, but generally speaking real security also requires
creating a chrooted filesystem topology, and a jail is still needed
to really segregate processes from each other. If you do so, however,
you can (for example) disable mount/umount and most global root-only
features.

* Add new system calls and a manual page for syscap_get(2) and syscap_set(2)

* Add sys/caps.h

* Add the "setcaps" userland utility and manual page.

* Remove priv.9 and the priv_check infrastructure, replacing it with
a newly designed caps infrastructure.

* The intention is to add path restriction lists and similar features to
improve jailess security in the near future, and to optimize the
priv_check code.

show more ...

dsynth - Add -f option for purge-distfiles

* purge-distfiles normally refuses to run if the dports tree has
Makefile or dependency errors, because that may cause it to
remove distfiles that are

dsynth - Add -f option for purge-distfiles

* purge-distfiles normally refuses to run if the dports tree has
Makefile or dependency errors, because that may cause it to
remove distfiles that are actually needed.

* Add the -f option to force it to run anyway.

show more ...

dsynth - Replace portsmon link by FreshPorts

dsynth - Fix bug, but actually keep the behavior

* Fix a bug where dexec was not copying the entry environment.
But actually, that is the behavior we want (a clean environment
prior to adding ou

dsynth - Fix bug, but actually keep the behavior

* Fix a bug where dexec was not copying the entry environment.
But actually, that is the behavior we want (a clean environment
prior to adding our own dsynth-managed environment) so then
comment out the code.

Reported-by: mjg

show more ...

dsynth - Add Numa_setsize option (defaults to disabled) (2)

* Add missing numa.c file

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are th

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are the only really viable choices,
otherwise the cpu mask will be so fragmented that even highly
parallel bulks are going to wind up with cpu threads left idle.

The NUMA partitioning is hacked at the moment but will generally
work with both AMD and Intel.

This works by round-robining N cpumask domains across available builder
slots. Insofar as DragonFlyBSD goes, this will localize per-process /
per-thread anonymous page allocations and kernel memory resources.

* However, DragonFlyBSD still spreads buffer cache pages (the file
cache) across all domains, including for tmpfs, The reason is that
such pages tend to be very long-lived and fully localizing them can
exhaust the VM page queues for related cpus and domains, forcing
contention inside vm_page_alloc() as multiple cpu threads contend
for fewer VM page queue slots.

show more ...

dsynth - Add section on per-port options

* Explain how per-port options work in the dsynth manual page.

dsynth - Ignore ports installed on the host during pkgscan

* Set LOCALBASE=<some_empty_dir> in the make commands issued by the
package scan so the host's installed packages are not probed.

Report

dsynth - Ignore ports installed on the host during pkgscan

* Set LOCALBASE=<some_empty_dir> in the make commands issued by the
package scan so the host's installed packages are not probed.

Reported-by: daftaupe

show more ...

dsynth - Implement a new 'add' directive

* This directive allows adding additional port(s) to the build list
of a running dsynth. It can be useful when the user does not wish
to interrupt the r

dsynth - Implement a new 'add' directive

* This directive allows adding additional port(s) to the build list
of a running dsynth. It can be useful when the user does not wish
to interrupt the running dsynth but wishes to add more ports and then
leave the dsynth unintended. For example because it might be half
way through building a huge package and interrupting it would be bad.

* dsynth will complete the current build as-per normal, including
rebuilding the repository if it would normally do so (without asking
interactively), and will then re-exec itself with the same options
and directive for the additional ports that were added in the interim.

show more ...

dsynth - Add Check_plist config file option

* dsynth -P turns on plist checking for builds. Note that this
feature is already automatically turned on when the 'everything'
and 'test' directives

dsynth - Add Check_plist config file option

* dsynth -P turns on plist checking for builds. Note that this
feature is already automatically turned on when the 'everything'
and 'test' directives are used.

* Add a Check_plist configuration file option that does the same thing
so -P does not have to be specified if the developer wants to always do
plist checking.

Requested-by: daftaupe

show more ...

dsynth: Call free() after asprintf() .

dsynth - better error reporting for dependency parsing (5)

Fix typo.

dsynth - better error reporting for dependency partsing (4)

* Add even more information. Print which dependency variable
is having the trouble.

dsynth - better error reporting for dependency partsing (3)

* Sigh

dsynth - better error reporting for dependency partsing (2)

* Output original dependency string (its big) prior to strsep
changes.

dsynth - add 'list-system' directive

* Add a 'list-system' directive which generates a list of installed
ports on the current machine that can then be used as a build list
on another machine.

dsynth - better error reporting for dependency partsing

* Do a better job reporting problems parsing dependencies

Reported-by: tuxillo

dsynth.1: Mark up hooks with .Pa (as they are file names).

Also, mark them up in the environment variable descriptions.

dsynth.1: Remove duplicated 'the'.

dsynth - Increase ncurses rate and other fields from 5 to 6 digits

* Increase displayed rate and other numerical fields from 5 to
6 digits. Certain dsynth actions (e.g. fetch-only) can now cause

dsynth - Increase ncurses rate and other fields from 5 to 6 digits

* Increase displayed rate and other numerical fields from 5 to
6 digits. Certain dsynth actions (e.g. fetch-only) can now cause
some of these numbers to exceed 5 digits.

* And for the rest, just future-proof it.

show more ...

dsynth - Add 'fetch-only {list/everything}*' directive (3)

* dsynth still follows dependency chains when fetching, which is
kinda what we want. However, failed fetches were blocking fetches
of

dsynth - Add 'fetch-only {list/everything}*' directive (3)

* dsynth still follows dependency chains when fetching, which is
kinda what we want. However, failed fetches were blocking fetches
of packages dependant on the failed one. Since we aren't building
in fetch-only mode, fake a 'success' so everything gets fetched that
can be fetched.

show more ...

dsynth - Protect threaded dbm_store() calls

* dsynth makes multi-threaded calls to dbm_store(), default
to throwing a mutex around the call to avoid dbm corruption.

(libc will also be fixed to

dsynth - Protect threaded dbm_store() calls

* dsynth makes multi-threaded calls to dbm_store(), default
to throwing a mutex around the call to avoid dbm corruption.

(libc will also be fixed to make dbm_*() calls thread-safe in
another commit, this is a catch-all).

show more ...

dsynth - Add 'fetch-only {list/everything}*' directive (2)

* Add to basic help output