Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their

Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their dependencies using
configuration data and the "dragonfly/base" dport found in
"/usr/src/dports.base".

This feature is intended to be used by the nrelease build in an upcoming
commit, and may also be used manually.

The /usr/dports tree must exist and contains the dports repo to use as the
basis of the dports build. It will be null-mounted in the chroot and the
"/usr/src/dports.base/dragonfly" directory will be null-mounted under it
to provide the "dragonfly/base" meta-port. The DPORTSBASE make variable
may be used to override the location of the main dports repo.

The build-all and install-all targets use a sophisticated chrooted
environment to do a completely clean dports build with dsynth. It
It will actually "make installworld" plus a clean "make distribution"
inside the chroot environment to create the environment, and the host
system's /etc/resolv.conf will be copied in.

build-all - Augmented when used WITH_DPORTS=YES
install-all - Augmented when used WITH_DPORTS=YES
buildportschroot - Build a fresh chroot environment for dsynth
mountports - Do all null mounts and cp's needed for the env
buildports - Run dsynth build dragonfly/base in the env
installports - Install the built ports on the host system (see note)
umountports - Remove the null mounts

WITH_DPORTS=YES - executes buildportschroot, mountports, buildports,
and umountports when run from build-all.

executes mountports, installports, umountports
when run from install-all.

DPORTSBASE=path - Defaults to /usr/dports, may be used to override
the location where the main dports tree can be
found.

NOTE: Installports may require interactivity as the base system might already
have installed ports. Generally speaking it is best to not revert any
ports you might already have installed, so feel free to tell it not to
install anything.

* Added various make targets and augmented build-all and install-all

* /usr/local/sbin added and the stale /usr/pkg/bin removed from _HOSTPATH.

* Add /usr/src/dports.base infrastructure.

It contains the dsynth.ini and pkg.conf files that will be used for the
local dsynth operations. Some paths are replaced with SED when copied
to the chroot. The dsynth.ini is relative to the chroot.

It also contains the dragonfly/base dport which is the meta-port that
dsynth will be told to build.

All other dports related files and directories are null-mounted from
/usr/dports.

* Add /usr/distfiles.base to mtree/BSD.usr.dist. This directory is also
created by the build if it does not exist since the host system might
not have run a new installworld recently.

This directory will be used during build-all (buildports) to store the
distfiles required to build dragonfly/base, so they only need to be
fetched once.

* Add the "-C ConfigBase" option to dsynth to make running it from the
chrooted environment easier. This overrides the /etc/dsynth directory
and removes /etc/dsynth as a fallback.

* dsynth conditionalizes calling syscap_set() for backwards compatibility
with older host systems that might not have been upgraded to include the
feature yet.

show more ...

kernel - Add per-process capability-based restrictions

* This new system allows userland to set capability restrictions which
turns off numerous kernel features and root accesses. These restricti

kernel - Add per-process capability-based restrictions

* This new system allows userland to set capability restrictions which
turns off numerous kernel features and root accesses. These restrictions
are inherited by sub-processes recursively. Once set, restrictions cannot
be removed.

Basic restrictions that mimic an unadorned jail can be enabled without
creating a jail, but generally speaking real security also requires
creating a chrooted filesystem topology, and a jail is still needed
to really segregate processes from each other. If you do so, however,
you can (for example) disable mount/umount and most global root-only
features.

* Add new system calls and a manual page for syscap_get(2) and syscap_set(2)

* Add sys/caps.h

* Add the "setcaps" userland utility and manual page.

* Remove priv.9 and the priv_check infrastructure, replacing it with
a newly designed caps infrastructure.

* The intention is to add path restriction lists and similar features to
improve jailess security in the near future, and to optimize the
priv_check code.

show more ...

dsynth - Add -f option for purge-distfiles

* purge-distfiles normally refuses to run if the dports tree has
Makefile or dependency errors, because that may cause it to
remove distfiles that are

dsynth - Add -f option for purge-distfiles

* purge-distfiles normally refuses to run if the dports tree has
Makefile or dependency errors, because that may cause it to
remove distfiles that are actually needed.

* Add the -f option to force it to run anyway.

show more ...

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are th

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are the only really viable choices,
otherwise the cpu mask will be so fragmented that even highly
parallel bulks are going to wind up with cpu threads left idle.

The NUMA partitioning is hacked at the moment but will generally
work with both AMD and Intel.

This works by round-robining N cpumask domains across available builder
slots. Insofar as DragonFlyBSD goes, this will localize per-process /
per-thread anonymous page allocations and kernel memory resources.

* However, DragonFlyBSD still spreads buffer cache pages (the file
cache) across all domains, including for tmpfs, The reason is that
such pages tend to be very long-lived and fully localizing them can
exhaust the VM page queues for related cpus and domains, forcing
contention inside vm_page_alloc() as multiple cpu threads contend
for fewer VM page queue slots.

show more ...

dsynth - Implement a new 'add' directive

* This directive allows adding additional port(s) to the build list
of a running dsynth. It can be useful when the user does not wish
to interrupt the r

dsynth - Implement a new 'add' directive

* This directive allows adding additional port(s) to the build list
of a running dsynth. It can be useful when the user does not wish
to interrupt the running dsynth but wishes to add more ports and then
leave the dsynth unintended. For example because it might be half
way through building a huge package and interrupting it would be bad.

* dsynth will complete the current build as-per normal, including
rebuilding the repository if it would normally do so (without asking
interactively), and will then re-exec itself with the same options
and directive for the additional ports that were added in the interim.

show more ...

dsynth - Add Check_plist config file option

* dsynth -P turns on plist checking for builds. Note that this
feature is already automatically turned on when the 'everything'
and 'test' directives

dsynth - Add Check_plist config file option

* dsynth -P turns on plist checking for builds. Note that this
feature is already automatically turned on when the 'everything'
and 'test' directives are used.

* Add a Check_plist configuration file option that does the same thing
so -P does not have to be specified if the developer wants to always do
plist checking.

Requested-by: daftaupe

show more ...

dsynth - add 'list-system' directive

* Add a 'list-system' directive which generates a list of installed
ports on the current machine that can then be used as a build list
on another machine.

dsynth - Add 'fetch-only {list/everything}*' directive (2)

* Add to basic help output

dsynth - Add 'fetch-only {list/everything}*' directive

* Implements a fetch-only feature which tells dsynth to fetch all
source distributions required to build the specified ports.
If 'everythin

dsynth - Add 'fetch-only {list/everything}*' directive

* Implements a fetch-only feature which tells dsynth to fetch all
source distributions required to build the specified ports.
If 'everything' is specified, the source distribuitons needed
to build the whole of dports will be fetched.

* Any source distributions already fetched are tested against their
checksum and re-fetched if necessary.

show more ...

dsynth - Add PKG_COMPRESSION_FORMAT

* Generate a PKG_COMPRESSION_FORMAT env variable in addition to
PKG_SUFX. This should fix the compression format used for
package generation to be the one sp

dsynth - Add PKG_COMPRESSION_FORMAT

* Generate a PKG_COMPRESSION_FORMAT env variable in addition to
PKG_SUFX. This should fix the compression format used for
package generation to be the one specified in dsynth.ini.

* For now keep PKG_SUFX in case dsynth is run on an older system.
It will cause dports to generate warnings in the logs, which can
be ignored.

show more ...

dsynth - Add -M scale option

* Add option to scale the number of workers allowed. The dynamic workers
calculation operates normally but all results are scaled accordingly.
Values between 0.01 a

dsynth - Add -M scale option

* Add option to scale the number of workers allowed. The dynamic workers
calculation operates normally but all results are scaled accordingly.
Values between 0.01 and 99.0 are acceptable.

This option does not modify the number of jobs for each worker.

* Typically used for load testing. Generally speaking, for actual use
the scale should be specified between 0.8 and 1.2. The default scale
factor is 1.0.

show more ...

dsynth: Delete obsolete packages upon 'prepare-system'

- Remove old versions of packages when doing 'prepare-system'
thus avoiding duplicate dependencies due to old versions still
being pr

dsynth: Delete obsolete packages upon 'prepare-system'

- Remove old versions of packages when doing 'prepare-system'
thus avoiding duplicate dependencies due to old versions still
being present along with newer ones.

show more ...

dsynth: Enable 'install' and 'deinstall' phases on test.

dsynth - Track contents of dports to detect changes

* Stat information and path names for the files making up a port
(in the dports directly tree) is rolled-up into a CRC and tracked
by dsynth.

dsynth - Track contents of dports to detect changes

* Stat information and path names for the files making up a port
(in the dports directly tree) is rolled-up into a CRC and tracked
by dsynth.

* By default, dsynth automatically rebuilds any dport that it detects
has changed, even if a binary package file already exists, including
any dependent ports.

* Add the -x and -xx options which may be used to override this behavior.

show more ...

dsynth - Automatically run at nice 10 by default

* dsynth now automatically nices sub-processes to +10. This is a
good middle-of-the-road nice to ensure that dsynth does not interfere
with othe

dsynth - Automatically run at nice 10 by default

* dsynth now automatically nices sub-processes to +10. This is a
good middle-of-the-road nice to ensure that dsynth does not interfere
with other system tasks too much (beyond the massive amount of memory
and cpu these builds burn normally).

We don't want to nice it too high or the scheduler won't have enough
room for its normal dynamic priority management.

* This feature may be disabled via -N.

show more ...

dsynth - Improve auto SlowStart a bit

* When MaxWorkers is >= 16, set the SlowStart at MaxWorkers / 4 instead
of at 1 so we do not have to wait forever for it to inch up to a
stable value.

dsynth - Delete obsolete packages for 'dsynth everything'

* When runnig 'dsynth everything', delete any obsolete packages
in the repo that don't show up in the dports scan.

Requested-by: zrj

dsynth - Abort on certain probe errors during pkglist scan

* If any directly-specified package or any direct dependency of
any directly-specified package fails to probe, we now abort
prior to en

dsynth - Abort on certain probe errors during pkglist scan

* If any directly-specified package or any direct dependency of
any directly-specified package fails to probe, we now abort
prior to entering ncurses mode.

* Note that 'dsynth everything' does not abort.

Requested-by: zrj

show more ...

dsynth - Adjust directive tests

* Adjust 'monitor' so it works as expected.

* 'version' does not need a working configuration to report.

dsynth - Add profile selection options

* Allow the profile to be specified via the DSYNTH_PROFILE env
variable or via the -p profile option, overriding the default
in /etc/dsynth.ini.

NOTE: I

dsynth - Add profile selection options

* Allow the profile to be specified via the DSYNTH_PROFILE env
variable or via the -p profile option, overriding the default
in /etc/dsynth.ini.

NOTE: If running more than one dsynth simultaneously on the same
machine, the following directives in dsynth.ini must be
different:

Directory_packages
Directory_repository
Directory_buildbase
Directory_logs

NOTE: The Directory_distfiles directive can be shared across
any number of running instances without running afoul
of colliding fetches.

* Allows easy selection from multiple profiles specified in
dsynth.ini.

* Create an active .lock file in the buildbase and complain if
another dsynth is running.

show more ...

dsynth - Fix 'debug' mode

* Fix the 'debug' directive to proper build pre-cursor packages
without freezing and to only build the packages explicitly specified
on the command line with freezing.

dsynth - Fix 'debug' mode

* Fix the 'debug' directive to proper build pre-cursor packages
without freezing and to only build the packages explicitly specified
on the command line with freezing.

* This reworks the global flag a bit. It wasn't entirely easy
because the WORKERs are exec'd sub-processes.

show more ...

dsynth - Parse flavors on command line, parse text file

* Parse flavor specifications on the command line.

* Allow a text file containing the list of packages to be specified
on the command line.

dsynth - Parse flavors on command line, parse text file

* Parse flavor specifications on the command line.

* Allow a text file containing the list of packages to be specified
on the command line.

* Make 'dsynth help' work even if no configuration file.

Reported-by: ftigeot

show more ...

dsynth - Improve 'debug' target, fix "x\n" in logs

* Improve the 'debug' target. Remove the target packages first,
just like the 'test' target, and include a description for the
'debug' target

dsynth - Improve 'debug' target, fix "x\n" in logs

* Improve the 'debug' target. Remove the target packages first,
just like the 'test' target, and include a description for the
'debug' target in dsynth help.

* Change the pty handshake to just a newline, it still shows up in the
logs due to (probably) master-pty echo mode, but it isn't really
noticeable verses the previous "x\n".

show more ...

dsynth - Add ARCH make var to bulk scan

* Add the ARCH make variable and env variable to bulk scan operations.

Reported-by: tuxillo

dsynth - Conditionalize use of check-plist stage

* Only run the check-plist stage for:

dsynth -P <anything>
dsynth -D everything
dsynth test ...