1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91; Meta_version= 2 (this is the default) 92Number_of_builders= 8 93Max_jobs_per_builder= 8 94Display_with_ncurses= true 95.Ed 96.Pp 97This places all major directories under 98.Pa /build/synth . 99If you want to use the same dports and the same distfiles as your base 100system, you can null-mount /usr/distfiles onto /build/synth/distfiles 101and /usr/dports onto /build/synth/dports with 102.Pa /etc/fstab 103entries as follows: 104.Bd -literal 105# Device Mountpoint FStype Options DumpPass# 106/usr/distfiles /build/synth/distfiles null rw 4 4 107/usr/dports /build/synth/dports null rw 4 4 108.Ed 109.Pp 110Please set the number of builders and the maximum number of jobs per 111builder according to available system resources. 112Remember that the total 113load on the system can be as high as (builders x jobs), and at least 4x 114that value in processes. 115Systems are typically restricted by memory and CPU horsepower. 116Start conservative and ramp up according to what your system can handle. 117A good rule of thumb is to set workers to the number of CPU threads your 118machine has or to 1/2 the number of gigabytes of memory your system has, 119whichever is lower. 120Then set the jobs per worker to no more than the 121number of CPU threads your machine has. 122.Pp 123.Nm 124has numerous features to manage machine load and swap usage to 125prevent a machine from being overloaded, allowing more workers 126to be configured than you might otherwise think is reasonable 127(which helps a lot when building the smaller ports). 128However, users running this program should be aware that very high loads 129and modest swap use are still likely to develop when building a large 130number of ports or when building very large ports like chromium. 131If the system is not dedicated to building packages you can reduce the 132impact to the rest of the system by running 133.Nm 134at nice +20 and also by reducing the number of workers and number of 135jobs per worker somewhat. 136.Pp 137We recommend that a minimum of 64GB of SSD-based swap be configured, 138or twice as much swap as main memory, whichever is the higher value. 139.Pp 140We recommend a minimum of 500GB of storage be configured in 141.Pa /build 142or wherever you have configured various directories. 143A full set of distfiles requires at least 120GB, a full dports including 144the git repo requires at least 1.5GB, and a full set of built packages 145requires at least 75GB. 146If using a filesystem such as HAMMER or HAMMER2 147which frees space overnight, double all of those numbers. 148.Pp 149The actual build infrastructure uses tmpfs... memory and swap, and does 150not use regular filesystem space. 151.Sh OPTIONS 152.Bl -tag -width indent 153.It Fl d[d...] 154Run in debug mode. 155If specified two or more times this will turn off 156ncurses and output the primary log (00_last_results.log) to the standard 157output, along with additional spew. 158.It Fl h 159Quickly output a synopsis of options and directives and exit. 160.It Fl m Ar gb 161Override the default package dependency memory target, in gigabytes. 162The default is 1/2 physical memory. 163The number of workers will be limited 164such that the aggregate size of package dependencies installed in each 165worker slot does not exceed this value. 166.Pp 167This handles a well-known effect where the sheer amount of data that has 168to be installed in tmpfs filesystems for large ports, when multiplied by 169the number of worker slots, can force excessive paging to occur and leave 170preciously little memory available to actually run compiles. 171Some paging 172is necessary to maintain maximum CPU utilization, but excessive paging 173can cause the whole machine to essentially become idle for extended 174periods of time. 175.It Fl M Ar scale 176Override dynamic workers calculation by a specific scale factor. 177Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 17820%, and 1.2 will increase the number of jobs by 20%. And so forth. 179.Pp 180This option may be specified in the range 0.01 to 99.0. Out of range values 181will simply be clipped. 182.It Fl p Ar profile 183Override the global profile default in 184.Pa /etc/dsynth/dsynth.ini , 185allowing you to trivially run whatever profile you like without having to 186edit the configuration file when switching. 187In addition, you can now run any number of dsynth's concurrently on the same 188machine without having to use a jail, each with a different profile, 189as long as the packages, repository, buildbase, and logs directories 190are different. 191.Pp 192Note that the distfiles directory can be shared and will not conflict 193or get confused with concurrent fetches. 194.It Fl s Ar n 195.Nm 196usually slow-starts the worker slots, beginning with one slot and increasing 197by one every 5 seconds until the maximum configured number of workers is 198reached. 199This gives 200.Nm 201a slower ramp that it can load manage against. 202Specifying 0 disables the slow-start feature and the maximum number of 203worker slots (limited by the dependency graph) will be loaded immediately. 204.It Fl v 205Quickly output the version and exit. 206.It Fl x 207.It Fl xx 208Normally dsynth builds a package for any of three reasons: (1) If the contents 209of the ports directory changes, (2) If anything the port depends on requires 210rebuilding so to will the port be rebuilt, (3) If there is no binary package 211already built for the port. 212.Pp 213If this option is specified, the first test is ignored. 214If this option is specified twice, the first and second tests are ignored. 215.It Fl y 216Automatically answer 'y'es to any questions. 217.It Fl D 218Turn on DEVELOPER mode when building ports. 219.It Fl P 220Include the check-plist stage. 221This is the default for the 222.Cm everything 223directive. 224.It Fl S[S] 225Turn off curses for script friendliness. 226The output will be log 00 and 227should be redirected to /dev/null or something similar. 228If you supply the options twice, color output escapes will also be 229turned off. 230You may also wish to use the 231.Fl y 232option for scripting dsynth. 233.It Fl N 234Normally 235.Nm 236nices its sub-processes to +10. 237This option disables the feature. 238.El 239.Sh DIRECTIVES 240Generally 241.Nm 242is run with a directive and some directives allow a list of ports to be 243specified. 244This list should be space-delimited in DIR/SUBDIR format, for example: 245.Ar www/chromium . 246For directives with an optional ports list, your current installed set 247of ports will be used if you do not specify a list. 248.Bl -tag -width indent 249.It Cm init 250Creates and initializes the 251.Pa /etc/dsynth 252directory if it does not exist. 253This directive will complain and exit if either 254.Pa /etc/dsynth 255or 256.Pa /usr/local/etc/dsynth 257exists. 258It will not create 259.Pa /etc/dsynth 260in this situation. 261.It Cm status 262This will do a dry-run of 263.Cm upgrade-system 264but not actually build anything. 265.It Cm cleanup 266This will clean up any left-over mounts from prior builds. 267.Nm 268attempts to clean up all processes and mounts when you interrupt 269a build but doesn't always succeed. 270.It Cm configure 271NOT CURRENTLY IMPLEMENTED 272.It Cm upgrade-system 273NOT CURRENTLY IMPLEMENTED. 274Incrementally build and upgrade your locally 275installed packages, then upgrade your local system with them. 276.It Cm prepare-system 277Incrementally build and upgrade your locally installed packages, but 278do not upgrade your system with them. 279.It Cm rebuild-repository 280Build or rebuild the database files for the configured repository. 281.It Cm purge-distfiles 282Delete any obsolete source distribution files. 283.It Cm reset-db 284Delete ports_crc.db from the build directory. 285This database is used to detect changes made to the dports tree. 286It will be regenerated on your next build without forcing any packages to be rebuilt. 287.It Cm status-everything 288This will do a dry-run of a full bulk build of everything, 289but not actually build anything. 290.It Cm everything 291This will build the entire dports tree and then rebuild the repository 292when it finishes. 293.It Cm version 294This is for synth compatibility. 295The version of 296.Nm 297will be printed and the program will exit. 298.It Cm help 299Output a synopsis of options and directives and exit. 300.It Cm status Op Ar ports 301Do a dry-run with 'build' of the given list. 302.It Cm build Op Ar ports 303Incrementally build dports based on the given list. 304When done, ask whether the repository should be rebuilt or not. 305.It Cm just-build Op Ar ports 306Incrementally build dports based on the given list, then 307exits. 308No post-build steps will be taken. 309.It Cm install Op Ar ports 310NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 311list (or using currently installed packages), then rebuild 312the repository and upgrade the system without asking any further 313questions. 314.It Cm force Op Ar ports 315This is the same as 'build' but will delete existing packages first. 316Dependencies are not deleted unless they are out of date. 317.It Cm test Op Ar ports 318This is the same as 'build' but sets the environment variable 319.Ev DEVELOPER 320to 321.Sq yes 322and pre-deletes specified packages. 323Dependencies are not deleted unless they are out of date. 324.It Cm debug Op Ar ports 325This is the same as 'build' but leaves the chroot mounts intact 326upon completion. 327.It Cm monitor Op Ar datfile 328Monitors a running dsynth instance. 329.El 330.Sh HOOKS 331.Nm 332provides several hooks that trigger at specific stages during the 333package building process. 334.Pp 335At the moment hooks are not configurable so the exact executable file is 336expected in the configuration directory with one of the names in the 337the list below. 338Hooks are run via 339.Xr execve 2 . 340.Bl -tag -width indent 341.It Cm hook_run_start 342This hook triggers when the overall build process starts. 343.It Cm hook_run_end 344This hook is called when the overall build process ends. 345.It Cm hook_pkg_success 346For each successful port built this hook will trigger. 347.It Cm hook_pkg_failure 348This hook will trigger for each port that fails to build. 349.It Cm hook_pkg_ignored 350Each port that is marked as ignored will make this hook to trigger. 351.It Cm hook_pkg_skipped 352Each skipped port will trigger this hook. 353.El 354.Pp 355A number of environment variables are available for hooks, always in the context 356of an ongoing build and within a specific configuration profile, unless 357overridden from the command-line. 358Some are only available for a specific hook. 359.Bl -tag -width DIR_REPOSITORY 360.It Ev PROFILE 361The configuration profile. 362.It Ev DIR_PACKAGES 363The packages base directory, i.e where index files are generated. 364.It Ev DIR_REPOSITORY 365The packages repository, where the actual package files are stored. 366.It Ev DIR_PORTS 367The ports directory. 368.It Ev DIR_OPTIONS 369The options directory. 370.It Ev DIR_DISTFILES 371The distfiles directory, where the distribution files are stored. 372.It Ev DIR_LOGS 373The logs directory, which is also where the html Report is generated. 374.It Ev DIR_BUILDBASE 375The build base directory. 376.It Ev PORTS_QUEUED 377The number of ports queued to be built (only for hook_run_start). 378.It Ev PORTS_BUILT 379The number of successfully built ports (only for hook_run_end). 380.It Ev PORTS_FAILED 381The number of ports for which the build failed (only for hook_run_end). 382.It Ev PORTS_IGNORED 383The number of ports that where ignored and, hence, not built 384(only for hook_run_end). 385.It Ev PORTS_SKIPPED 386The number of ports that were skipped in the build (only for hook_run_end). 387.It Ev RESULT 388The result (success, failure, ignored, skipped) for the build of an individual 389port (only for port specific hooks). 390.It Ev ORIGIN 391The origin of a port (only for port specific hooks). 392.It Ev FLAVOR 393The flavor of a port (only for port specific hooks). 394.It Ev PKGNAME 395The port name (only for port specific hooks). 396.El 397.Sh FILES 398.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 399.It Pa /etc/dsynth/dsynth.ini 400The primary configuration file. 401If not found, 402.Nm 403will also look in 404.Pa /usr/local/etc/dsynth/dsynth.ini . 405.Pp 406.It Pa /etc/dsynth/LiveSystem-make.conf 407Typically contains the environment variables that will be set in 408the workers. 409.Nm 410firewalls the environment it is run under from the environment it 411provides to the workers. 412.Pp 413.It Pa /build/synth/build 414Recommended setting for 415.Va Directory_buildbase , 416contains the build infrastructure... typically a template, mirrored 417system directories, and mount points for all the worker slots. 418The template will be [re]generated if 'pkg' needs to be built or 419if the 420.Pa .template.good 421file in this directory is deleted. 422.Pp 423.It Pa /build/synth/distfiles 424Recommended setting for 425.Va Directory_distfiles , 426ports to a directory into which 427.Nm 428will download any source distribution files required for building. 429.Pp 430.It Pa /build/synth/dports 431Recommended setting for 432.Va Directory_portsdir , 433points to a checked out dports repo. 434Note that 435.Nm 436does not automatically 'git pull' or otherwise synchronize the dports repo, 437you must do that yourself prior to starting a build. 438.Pp 439.It Pa /build/synth/live_packages 440Recommended setting for 441.Va Directory_packages , 442points to a directory which will contain the completed application 443packages. 444.Pp 445.It Pa /build/synth/logs 446Recommended setting for 447.Va Directory_logs , 448all log files will be placed in this directory. 449Special management logfiles begin with the numeral '0' for easily 450location. 451The logfiles for ports while and after building are stored in the 452form subdir____portname.log, with three underscores. 453.Pp 454.It Pa /build/synth/options 455Recommended setting for 456.Va Directory_options , 457where options overrides for specific ports may be located. 458.Pp 459.It Pa / 460Recommended setting for 461.Va Directory_system , 462which 463.Nm 464uses as a basis for creating the jails or chroots in each worker slot 465during building. 466No part of the system root is ever NULL-mounted read-write... it is always 467NULL-mounted read-only. 468Some elements from the system base will be mirrored in the build-base 469as an optimization. 470.Pp 471Note that the packages directory and the distfiles directory is mounted 472read-write in jails or chroots. 473All other r/w filesystems in the workers are 474.Xr tmpfs 5 475based filesystems and will be created and torn-down for each port. 476.Pp 477.It Pa .txz 478.It Pa .tgz 479.It Pa .tar 480.It Pa .tbz 481.It Pa .tzst 482The recommended setting for 483.Va Package_suffix 484is either 485.Pa .txz 486or 487.Pa .tgz . 488Use 489.Pa .txz 490for better compression at the cost of somewhat slower bulk builds due 491to the time overhead for compression and decompression, or 492use 493.Pa .tgz 494for modest compression and very fast compression and decompression. 495Due to the way the builder works, package dependencies are fresly 496installed into the chroot slot for each package being built, so 497decompression time matters. 498.Pp 499.It 2 500The default setting for 501.Va Meta_version 502is now 2. You can override it with this configuration variable. 503.El 504.Sh EXIT STATUS 505.Ex -std 506.Sh SEE ALSO 507.Xr synth 1 , 508.Xr dports 7 509.Sh HISTORY 510The 511.Nm 512utility first appeared in 513.Dx 5.7 . 514.Sh AUTHORS 515.An Matthew Dillon Aq Mt dillon@backplane.com 516