Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their

Add integrated dports build feature to build-all and install-all targets

When WITH_DPORTS=YES is used build-all and install-all, these targets will
also build the base set of ports and all of their dependencies using
configuration data and the "dragonfly/base" dport found in
"/usr/src/dports.base".

This feature is intended to be used by the nrelease build in an upcoming
commit, and may also be used manually.

The /usr/dports tree must exist and contains the dports repo to use as the
basis of the dports build. It will be null-mounted in the chroot and the
"/usr/src/dports.base/dragonfly" directory will be null-mounted under it
to provide the "dragonfly/base" meta-port. The DPORTSBASE make variable
may be used to override the location of the main dports repo.

The build-all and install-all targets use a sophisticated chrooted
environment to do a completely clean dports build with dsynth. It
It will actually "make installworld" plus a clean "make distribution"
inside the chroot environment to create the environment, and the host
system's /etc/resolv.conf will be copied in.

build-all - Augmented when used WITH_DPORTS=YES
install-all - Augmented when used WITH_DPORTS=YES
buildportschroot - Build a fresh chroot environment for dsynth
mountports - Do all null mounts and cp's needed for the env
buildports - Run dsynth build dragonfly/base in the env
installports - Install the built ports on the host system (see note)
umountports - Remove the null mounts

WITH_DPORTS=YES - executes buildportschroot, mountports, buildports,
and umountports when run from build-all.

executes mountports, installports, umountports
when run from install-all.

DPORTSBASE=path - Defaults to /usr/dports, may be used to override
the location where the main dports tree can be
found.

NOTE: Installports may require interactivity as the base system might already
have installed ports. Generally speaking it is best to not revert any
ports you might already have installed, so feel free to tell it not to
install anything.

* Added various make targets and augmented build-all and install-all

* /usr/local/sbin added and the stale /usr/pkg/bin removed from _HOSTPATH.

* Add /usr/src/dports.base infrastructure.

It contains the dsynth.ini and pkg.conf files that will be used for the
local dsynth operations. Some paths are replaced with SED when copied
to the chroot. The dsynth.ini is relative to the chroot.

It also contains the dragonfly/base dport which is the meta-port that
dsynth will be told to build.

All other dports related files and directories are null-mounted from
/usr/dports.

* Add /usr/distfiles.base to mtree/BSD.usr.dist. This directory is also
created by the build if it does not exist since the host system might
not have run a new installworld recently.

This directory will be used during build-all (buildports) to store the
distfiles required to build dragonfly/base, so they only need to be
fetched once.

* Add the "-C ConfigBase" option to dsynth to make running it from the
chrooted environment easier. This overrides the /etc/dsynth directory
and removes /etc/dsynth as a fallback.

* dsynth conditionalizes calling syscap_set() for backwards compatibility
with older host systems that might not have been upgraded to include the
feature yet.

show more ...

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are th

dsynth - Add Numa_setsize option (defaults to disabled)

* Currently only for testing purposes and not recommended for general
use. Default is 0 (disabled).

* If enabled, a value of 2 or 4 are the only really viable choices,
otherwise the cpu mask will be so fragmented that even highly
parallel bulks are going to wind up with cpu threads left idle.

The NUMA partitioning is hacked at the moment but will generally
work with both AMD and Intel.

This works by round-robining N cpumask domains across available builder
slots. Insofar as DragonFlyBSD goes, this will localize per-process /
per-thread anonymous page allocations and kernel memory resources.

* However, DragonFlyBSD still spreads buffer cache pages (the file
cache) across all domains, including for tmpfs, The reason is that
such pages tend to be very long-lived and fully localizing them can
exhaust the VM page queues for related cpus and domains, forcing
contention inside vm_page_alloc() as multiple cpu threads contend
for fewer VM page queue slots.

show more ...

dsynth - Track contents of dports to detect changes

* Stat information and path names for the files making up a port
(in the dports directly tree) is rolled-up into a CRC and tracked
by dsynth.

dsynth - Track contents of dports to detect changes

* Stat information and path names for the files making up a port
(in the dports directly tree) is rolled-up into a CRC and tracked
by dsynth.

* By default, dsynth automatically rebuilds any dport that it detects
has changed, even if a binary package file already exists, including
any dependent ports.

* Add the -x and -xx options which may be used to override this behavior.

show more ...

dsynth - Add HTML support

* Bring in progress.css, progress.html, progress.js, and icons from
the synth project. Add John's copyright to progress.js.

* dsynth now generates the logs/Report subdi

dsynth - Add HTML support

* Bring in progress.css, progress.html, progress.js, and icons from
the synth project. Add John's copyright to progress.js.

* dsynth now generates the logs/Report subdirectory, all necessary
files, and dynamically generates the json.

The summary.json database is updated every 10 seconds to reduce wear and
tear.

* Split the history files in 50KB segments to reduce network traffic.

show more ...

phantasia(6)/dsynth(1): Add two missing DPADD definitions.

dsynth - Enhance purge-distfiles directive

* Improve the distfiles list by using -VALLFILES instead of -VDISTFILES.

* Match up the related lock files so we can purge lock files that
are no longer

dsynth - Enhance purge-distfiles directive

* Improve the distfiles list by using -VALLFILES instead of -VDISTFILES.

* Match up the related lock files so we can purge lock files that
are no longer relevant.

show more ...

dsynth(1): Fix DPADD situation in the Makefile.

dsynth(1): Fix 'make install' with $DESTDIR set.

Now that the creation of /usr/share/dsynth is in the mtree file
(see 652cecd95046a092589807c7eea9799b71530ca4) and given that dsynth
is hooked into t

dsynth(1): Fix 'make install' with $DESTDIR set.

Now that the creation of /usr/share/dsynth is in the mtree file
(see 652cecd95046a092589807c7eea9799b71530ca4) and given that dsynth
is hooked into the build too, we could remove the beforeinstall
target, but it might still be useful for people running release,
for example. So keep it instead and fix it for when $DESTDIR is set.

show more ...

dsynth - Abstract the run-status infrastructure

* Abstract the run-status infrastructure and add skeletons for html
and a 'dsynth monitor' style monitor.

Neither is implemented yet but this mov

dsynth - Abstract the run-status infrastructure

* Abstract the run-status infrastructure and add skeletons for html
and a 'dsynth monitor' style monitor.

Neither is implemented yet but this moves most of the hard work
out of gui.c (and renames it to ncurses.c) so it doesn't have to
be replicated for the other two.

show more ...

dsynth - Add manual page, change configuration directory, more

* Change the configuration location from /usr/local/etc/synth.ini
to /etc/dsynth/dsynth.ini and (as an alternate)
/usr/local/etc/ds

dsynth - Add manual page, change configuration directory, more

* Change the configuration location from /usr/local/etc/synth.ini
to /etc/dsynth/dsynth.ini and (as an alternate)
/usr/local/etc/dsynth/dsynth.ini

* Add 'dsynth debug <port>' directive. This will build all
dependencies and then build the specified port and freeze
its worker slot with mounts still intact.

* Add the 'dsynth init' directive. This will initialize the
/etc/dsynth directory and default configuration files.
This directive will refuse to do anything if /etc/dsynth
or /usr/local/etc/dsynth already exists.

* Allow 'Number_of_builders' and/or 'Max_jobs_per_builder' to
be specified as 0. If 0, these parameters will be
auto-configured.

* Add the dsynth(1) manual page.

* Include a procfs (/proc) mount in the worker chroot.

* Change a 'cpdup' to a 'cp' to improve portability. Also change
how the Template is generated a bit.

show more ...

dsynth - Fleshout functions

* Add dynamic MaxWorker reduction based on the load. The number of
workers will be reduced by (load * 2 / ncpus) when the load
exceeds (ncpus * 2.0), with a floor of

dsynth - Fleshout functions

* Add dynamic MaxWorker reduction based on the load. The number of
workers will be reduced by (load * 2 / ncpus) when the load
exceeds (ncpus * 2.0), with a floor of min(4, MaxWorkers).

The idea here is to reduce unnecessary swap use when building
larger packages that push the load up greatly.

* Clean-up terminal spam after ncurses has been turned on.

* Generate fatal error if required configured directories are
missing.

* Add -y (yes to all) option.

* Implement 'status' and 'status-everything'

* Implement 'purge-distfiles' (initial work)

* In scans that dump percentages, print 100% at the end before
moving onto the next step.

show more ...

dsynth - Initial commit (unhooked from buildworld)

* DSynth is basically synth written in C, from scratch. It is designed
to give us a bulk builder in base and be friendly to porting and jails

dsynth - Initial commit (unhooked from buildworld)

* DSynth is basically synth written in C, from scratch. It is designed
to give us a bulk builder in base and be friendly to porting and jails
down the line (for now its uses chroot's).

The original synth was written by John R. Marino and its basic flow
was used in writing this program, but as it was written in ada no
code was directly copied.

* The intent is to make dsynth compatible with synth's configuration
files and directory structure.

* This is a work in progress and not yet ready for prime-time. Pushing
so we can get some more eyeballs. Most of the directives do not yet
work (everything, and build works, and 'cleanup' can be used to clean
up any dangling mounts).

* Not connected to the build yet. A great deal more work is needed.
As-of this commit, these areas need work:

- Uname/architecture/release-versioning stuff.

- Possibly additional environment variables or make variables
for certain cases and situations.

- The repo building step.

- Web interface.

- Dynamic load management (reduce the number of workers dynamically
based on load/ncpus).

- curses cleanup and window resizing.

show more ...