1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91Number_of_builders= 8 92Max_jobs_per_builder= 8 93Display_with_ncurses= true 94.Ed 95.Pp 96This places all major directories under 97.Pa /build/synth . 98If you want to use the same dports and the same distfiles as your base 99system, you can null-mount /usr/distfiles onto /build/synth/distfiles 100and /usr/dports onto /build/synth/dports with 101.Pa /etc/fstab 102entries as follows: 103.Bd -literal 104# Device Mountpoint FStype Options DumpPass# 105/usr/distfiles /build/synth/distfiles null rw 4 4 106/usr/dports /build/synth/dports null rw 4 4 107.Ed 108.Pp 109Please set the number of builders and the maximum number of jobs per 110builder according to available system resources. 111Remember that the total 112load on the system can be as high as (builders x jobs), and at least 4x 113that value in processes. 114Systems are typically restricted by memory and CPU horsepower. 115Start conservative and ramp up according to what your system can handle. 116A good rule of thumb is to set workers to the number of CPU threads your 117machine has or to 1/2 the number of gigabytes of memory your system has, 118whichever is lower. 119Then set the jobs per worker to no more than the 120number of CPU threads your machine has. 121.Pp 122.Nm 123has numerous features to manage machine load and swap usage to 124prevent a machine from being overloaded, allowing more workers 125to be configured than you might otherwise think is reasonable 126(which helps a lot when building the smaller ports). 127However, users running this program should be aware that very high loads 128and modest swap use are still likely to develop when building a large 129number of ports or when building very large ports like chromium. 130If the system is not dedicated to building packages you can reduce the 131impact to the rest of the system by running 132.Nm 133at nice +20 and also by reducing the number of workers and number of 134jobs per worker somewhat. 135.Pp 136We recommend that a minimum of 64GB of SSD-based swap be configured, 137or twice as much swap as main memory, whichever is the higher value. 138.Pp 139We recommend a minimum of 500GB of storage be configured in 140.Pa /build 141or wherever you have configured various directories. 142A full set of distfiles requires at least 120GB, a full dports including 143the git repo requires at least 1.5GB, and a full set of built packages 144requires at least 75GB. 145If using a filesystem such as HAMMER or HAMMER2 146which frees space overnight, double all of those numbers. 147.Pp 148The actual build infrastructure uses tmpfs... memory and swap, and does 149not use regular filesystem space. 150.Sh OPTIONS 151.Bl -tag -width indent 152.It Fl d[d...] 153Run in debug mode. 154If specified two or more times this will turn off 155ncurses and output the primary log (00_last_results.log) to the standard 156output, along with additional spew. 157.It Fl h 158Quickly output a synopsis of options and directives and exit. 159.It Fl m Ar gb 160Override the default package dependency memory target, in gigabytes. 161The default is 1/2 physical memory. 162The number of workers will be limited 163such that the aggregate size of package dependencies installed in each 164worker slot does not exceed this value. 165.Pp 166This handles a well-known effect where the sheer amount of data that has 167to be installed in tmpfs filesystems for large ports, when multiplied by 168the number of worker slots, can force excessive paging to occur and leave 169preciously little memory available to actually run compiles. 170Some paging 171is necessary to maintain maximum CPU utilization, but excessive paging 172can cause the whole machine to essentially become idle for extended 173periods of time. 174.It Fl M Ar scale 175Override dynamic workers calculation by a specific scale factor. 176Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 17720%, and 1.2 will increase the number of jobs by 20%. And so forth. 178.Pp 179This option may be specified in the range 0.01 to 99.0. Out of range values 180will simply be clipped. 181.It Fl p Ar profile 182Override the global profile default in 183.Pa /etc/dsynth/dsynth.ini , 184allowing you to trivially run whatever profile you like without having to 185edit the configuration file when switching. 186In addition, you can now run any number of dsynth's concurrently on the same 187machine without having to use a jail, each with a different profile, 188as long as the packages, repository, buildbase, and logs directories 189are different. 190.Pp 191Note that the distfiles directory can be shared and will not conflict 192or get confused with concurrent fetches. 193.It Fl s Ar n 194.Nm 195usually slow-starts the worker slots, beginning with one slot and increasing 196by one every 5 seconds until the maximum configured number of workers is 197reached. 198This gives 199.Nm 200a slower ramp that it can load manage against. 201Specifying 0 disables the slow-start feature and the maximum number of 202worker slots (limited by the dependency graph) will be loaded immediately. 203.It Fl v 204Quickly output the version and exit. 205.It Fl x 206.It Fl xx 207Normally dsynth builds a package for any of three reasons: (1) If the contents 208of the ports directory changes, (2) If anything the port depends on requires 209rebuilding so to will the port be rebuilt, (3) If there is no binary package 210already built for the port. 211.Pp 212If this option is specified, the first test is ignored. 213If this option is specified twice, the first and second tests are ignored. 214.It Fl y 215Automatically answer 'y'es to any questions. 216.It Fl D 217Turn on DEVELOPER mode when building ports. 218.It Fl P 219Include the check-plist stage. 220This is the default for the 221.Cm everything 222directive. 223.It Fl S[S] 224Turn off curses for script friendliness. 225The output will be log 00 and 226should be redirected to /dev/null or something similar. 227If you supply the options twice, color output escapes will also be 228turned off. 229You may also wish to use the 230.Fl y 231option for scripting dsynth. 232.It Fl N 233Normally 234.Nm 235nices its sub-processes to +10. 236This option disables the feature. 237.El 238.Sh DIRECTIVES 239Generally 240.Nm 241is run with a directive and some directives allow a list of ports to be 242specified. 243This list should be space-delimited in DIR/SUBDIR format, for example: 244.Ar www/chromium . 245For directives with an optional ports list, your current installed set 246of ports will be used if you do not specify a list. 247.Bl -tag -width indent 248.It Cm init 249Creates and initializes the 250.Pa /etc/dsynth 251directory if it does not exist. 252This directive will complain and exit if either 253.Pa /etc/dsynth 254or 255.Pa /usr/local/etc/dsynth 256exists. 257It will not create 258.Pa /etc/dsynth 259in this situation. 260.It Cm status 261This will do a dry-run of 262.Cm upgrade-system 263but not actually build anything. 264.It Cm cleanup 265This will clean up any left-over mounts from prior builds. 266.Nm 267attempts to clean up all processes and mounts when you interrupt 268a build but doesn't always succeed. 269.It Cm configure 270NOT CURRENTLY IMPLEMENTED 271.It Cm upgrade-system 272NOT CURRENTLY IMPLEMENTED. 273Incrementally build and upgrade your locally 274installed packages, then upgrade your local system with them. 275.It Cm prepare-system 276Incrementally build and upgrade your locally installed packages, but 277do not upgrade your system with them. 278.It Cm rebuild-repository 279Build or rebuild the database files for the configured repository. 280.It Cm purge-distfiles 281Delete any obsolete source distribution files. 282.It Cm reset-db 283Delete ports_crc.db from the build directory. 284This database is used to detect changes made to the dports tree. 285It will be regenerated on your next build without forcing any packages to be rebuilt. 286.It Cm status-everything 287This will do a dry-run of a full bulk build of everything, 288but not actually build anything. 289.It Cm everything 290This will build the entire dports tree and then rebuild the repository 291when it finishes. 292.It Cm version 293This is for synth compatibility. 294The version of 295.Nm 296will be printed and the program will exit. 297.It Cm help 298Output a synopsis of options and directives and exit. 299.It Cm status Op Ar ports 300Do a dry-run with 'build' of the given list. 301.It Cm build Op Ar ports 302Incrementally build dports based on the given list. 303When done, ask whether the repository should be rebuilt or not. 304.It Cm just-build Op Ar ports 305Incrementally build dports based on the given list, then 306exits. 307No post-build steps will be taken. 308.It Cm install Op Ar ports 309NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 310list (or using currently installed packages), then rebuild 311the repository and upgrade the system without asking any further 312questions. 313.It Cm force Op Ar ports 314This is the same as 'build' but will delete existing packages first. 315Dependencies are not deleted unless they are out of date. 316.It Cm test Op Ar ports 317This is the same as 'build' but sets the environment variable 318.Ev DEVELOPER 319to 320.Sq yes 321and pre-deletes specified packages. 322Dependencies are not deleted unless they are out of date. 323.It Cm debug Op Ar ports 324This is the same as 'build' but leaves the chroot mounts intact 325upon completion. 326.It Cm monitor Op Ar datfile 327Monitors a running dsynth instance. 328.El 329.Sh HOOKS 330.Nm 331provides several hooks that trigger at specific stages during the 332package building process. 333.Pp 334At the moment hooks are not configurable so the exact executable file is 335expected in the configuration directory with one of the names in the 336the list below. 337Hooks are run via 338.Xr execve 2 . 339.Bl -tag -width indent 340.It Cm hook_run_start 341This hook triggers when the overall build process starts. 342.It Cm hook_run_end 343This hook is called when the overall build process ends. 344.It Cm hook_pkg_success 345For each successful port built this hook will trigger. 346.It Cm hook_pkg_failure 347This hook will trigger for each port that fails to build. 348.It Cm hook_pkg_ignored 349Each port that is marked as ignored will make this hook to trigger. 350.It Cm hook_pkg_skipped 351Each skipped port will trigger this hook. 352.El 353.Pp 354A number of environment variables are available for hooks, always in the context 355of an ongoing build and within a specific configuration profile, unless 356overridden from the command-line. 357Some are only available for a specific hook. 358.Bl -tag -width DIR_REPOSITORY 359.It Ev PROFILE 360The configuration profile. 361.It Ev DIR_PACKAGES 362The packages base directory, i.e where index files are generated. 363.It Ev DIR_REPOSITORY 364The packages repository, where the actual package files are stored. 365.It Ev DIR_PORTS 366The ports directory. 367.It Ev DIR_OPTIONS 368The options directory. 369.It Ev DIR_DISTFILES 370The distfiles directory, where the distribution files are stored. 371.It Ev DIR_LOGS 372The logs directory, which is also where the html Report is generated. 373.It Ev DIR_BUILDBASE 374The build base directory. 375.It Ev PORTS_QUEUED 376The number of ports queued to be built (only for hook_run_start). 377.It Ev PORTS_BUILT 378The number of successfully built ports (only for hook_run_end). 379.It Ev PORTS_FAILED 380The number of ports for which the build failed (only for hook_run_end). 381.It Ev PORTS_IGNORED 382The number of ports that where ignored and, hence, not built 383(only for hook_run_end). 384.It Ev PORTS_SKIPPED 385The number of ports that were skipped in the build (only for hook_run_end). 386.It Ev RESULT 387The result (success, failure, ignored, skipped) for the build of an individual 388port (only for port specific hooks). 389.It Ev ORIGIN 390The origin of a port (only for port specific hooks). 391.It Ev FLAVOR 392The flavor of a port (only for port specific hooks). 393.It Ev PKGNAME 394The port name (only for port specific hooks). 395.El 396.Sh FILES 397.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 398.It Pa /etc/dsynth/dsynth.ini 399The primary configuration file. 400If not found, 401.Nm 402will also look in 403.Pa /usr/local/etc/dsynth/dsynth.ini . 404.Pp 405.It Pa /etc/dsynth/LiveSystem-make.conf 406Typically contains the environment variables that will be set in 407the workers. 408.Nm 409firewalls the environment it is run under from the environment it 410provides to the workers. 411.Pp 412.It Pa /build/synth/build 413Recommended setting for 414.Va Directory_buildbase , 415contains the build infrastructure... typically a template, mirrored 416system directories, and mount points for all the worker slots. 417The template will be [re]generated if 'pkg' needs to be built or 418if the 419.Pa .template.good 420file in this directory is deleted. 421.Pp 422.It Pa /build/synth/distfiles 423Recommended setting for 424.Va Directory_distfiles , 425ports to a directory into which 426.Nm 427will download any source distribution files required for building. 428.Pp 429.It Pa /build/synth/dports 430Recommended setting for 431.Va Directory_portsdir , 432points to a checked out dports repo. 433Note that 434.Nm 435does not automatically 'git pull' or otherwise synchronize the dports repo, 436you must do that yourself prior to starting a build. 437.Pp 438.It Pa /build/synth/live_packages 439Recommended setting for 440.Va Directory_packages , 441points to a directory which will contain the completed application 442packages. 443.Pp 444.It Pa /build/synth/logs 445Recommended setting for 446.Va Directory_logs , 447all log files will be placed in this directory. 448Special management logfiles begin with the numeral '0' for easily 449location. 450The logfiles for ports while and after building are stored in the 451form subdir____portname.log, with three underscores. 452.Pp 453.It Pa /build/synth/options 454Recommended setting for 455.Va Directory_options , 456where options overrides for specific ports may be located. 457.Pp 458.It Pa / 459Recommended setting for 460.Va Directory_system , 461which 462.Nm 463uses as a basis for creating the jails or chroots in each worker slot 464during building. 465No part of the system root is ever NULL-mounted read-write... it is always 466NULL-mounted read-only. 467Some elements from the system base will be mirrored in the build-base 468as an optimization. 469.Pp 470Note that the packages directory and the distfiles directory is mounted 471read-write in jails or chroots. 472All other r/w filesystems in the workers are 473.Xr tmpfs 5 474based filesystems and will be created and torn-down for each port. 475.Pp 476.It Pa .txz 477.It Pa .tgz 478.It Pa .tar 479.It Pa .tbz 480.It Pa .tzst 481The recommended setting for 482.Va Package_suffix 483is either 484.Pa .txz 485or 486.Pa .tgz . 487Use 488.Pa .txz 489for better compression at the cost of somewhat slower bulk builds due 490to the time overhead for compression and decompression, or 491use 492.Pa .tgz 493for modest compression and very fast compression and decompression. 494Due to the way the builder works, package dependencies are fresly 495installed into the chroot slot for each package being built, so 496decompression time matters. 497.El 498.Sh EXIT STATUS 499.Ex -std 500.Sh SEE ALSO 501.Xr synth 1 , 502.Xr dports 7 503.Sh HISTORY 504The 505.Nm 506utility first appeared in 507.Dx 5.7 . 508.Sh AUTHORS 509.An Matthew Dillon Aq Mt dillon@backplane.com 510