1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91; Meta_version= 2 (this is the default) 92; Check_plist= false (this is the default) 93Number_of_builders= 8 94Max_jobs_per_builder= 8 95Display_with_ncurses= true 96.Ed 97.Pp 98This places all major directories under 99.Pa /build/synth . 100If you want to use the same dports and the same distfiles as your base 101system, you can null-mount /usr/distfiles onto /build/synth/distfiles 102and /usr/dports onto /build/synth/dports with 103.Pa /etc/fstab 104entries as follows: 105.Bd -literal 106# Device Mountpoint FStype Options DumpPass# 107/usr/distfiles /build/synth/distfiles null rw 4 4 108/usr/dports /build/synth/dports null rw 4 4 109.Ed 110.Pp 111Please set the number of builders and the maximum number of jobs per 112builder according to available system resources. 113Remember that the total 114load on the system can be as high as (builders x jobs), and at least 4x 115that value in processes. 116Systems are typically restricted by memory and CPU horsepower. 117Start conservative and ramp up according to what your system can handle. 118A good rule of thumb is to set workers to the number of CPU threads your 119machine has or to 1/2 the number of gigabytes of memory your system has, 120whichever is lower. 121Then set the jobs per worker to no more than the 122number of CPU threads your machine has. 123.Pp 124.Nm 125has numerous features to manage machine load and swap usage to 126prevent a machine from being overloaded, allowing more workers 127to be configured than you might otherwise think is reasonable 128(which helps a lot when building the smaller ports). 129However, users running this program should be aware that very high loads 130and modest swap use are still likely to develop when building a large 131number of ports or when building very large ports like chromium. 132If the system is not dedicated to building packages you can reduce the 133impact to the rest of the system by running 134.Nm 135at nice +20 and also by reducing the number of workers and number of 136jobs per worker somewhat. 137.Pp 138We recommend that a minimum of 64GB of SSD-based swap be configured, 139or twice as much swap as main memory, whichever is the higher value. 140.Pp 141We recommend a minimum of 500GB of storage be configured in 142.Pa /build 143or wherever you have configured various directories. 144A full set of distfiles requires at least 120GB, a full dports including 145the git repo requires at least 1.5GB, and a full set of built packages 146requires at least 75GB. 147If using a filesystem such as HAMMER or HAMMER2 148which frees space overnight, double all of those numbers. 149.Pp 150The actual build infrastructure uses tmpfs... memory and swap, and does 151not use regular filesystem space. 152.Sh OPTIONS 153.Bl -tag -width indent 154.It Fl d[d...] 155Run in debug mode. 156If specified two or more times this will turn off 157ncurses and output the primary log (00_last_results.log) to the standard 158output, along with additional spew. 159.It Fl h 160Quickly output a synopsis of options and directives and exit. 161.It Fl m Ar gb 162Override the default package dependency memory target, in gigabytes. 163The default is 1/2 physical memory. 164The number of workers will be limited 165such that the aggregate size of package dependencies installed in each 166worker slot does not exceed this value. 167.Pp 168This handles a well-known effect where the sheer amount of data that has 169to be installed in tmpfs filesystems for large ports, when multiplied by 170the number of worker slots, can force excessive paging to occur and leave 171preciously little memory available to actually run compiles. 172Some paging 173is necessary to maintain maximum CPU utilization, but excessive paging 174can cause the whole machine to essentially become idle for extended 175periods of time. 176.It Fl M Ar scale 177Override dynamic workers calculation by a specific scale factor. 178Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 17920%, and 1.2 will increase the number of jobs by 20%. And so forth. 180.Pp 181This option may be specified in the range 0.01 to 99.0. Out of range values 182will simply be clipped. 183.It Fl p Ar profile 184Override the global profile default in 185.Pa /etc/dsynth/dsynth.ini , 186allowing you to trivially run whatever profile you like without having to 187edit the configuration file when switching. 188In addition, you can now run any number of dsynth's concurrently on the same 189machine without having to use a jail, each with a different profile, 190as long as the packages, repository, buildbase, and logs directories 191are different. 192.Pp 193Note that the distfiles directory can be shared and will not conflict 194or get confused with concurrent fetches. 195.It Fl s Ar n 196.Nm 197usually slow-starts the worker slots, beginning with one slot and increasing 198by one every 5 seconds until the maximum configured number of workers is 199reached. 200This gives 201.Nm 202a slower ramp that it can load manage against. 203Specifying 0 disables the slow-start feature and the maximum number of 204worker slots (limited by the dependency graph) will be loaded immediately. 205.It Fl v 206Quickly output the version and exit. 207.It Fl x 208.It Fl xx 209Normally dsynth builds a package for any of three reasons: (1) If the contents 210of the ports directory changes, (2) If anything the port depends on requires 211rebuilding so to will the port be rebuilt, (3) If there is no binary package 212already built for the port. 213.Pp 214If this option is specified, the first test is ignored. 215If this option is specified twice, the first and second tests are ignored. 216.It Fl y 217Automatically answer 'y'es to any questions. 218.It Fl D 219Turn on DEVELOPER mode when building ports. 220.It Fl P 221Include the check-plist stage. 222This is the default for the 223.Cm everything 224and 225.Cm test 226directives. 227This feature may also be turned on via the 228.Va Check_plist 229option in the configuration file. 230.It Fl S[S] 231Turn off curses for script friendliness. 232The output will be log 00 and 233should be redirected to /dev/null or something similar. 234If you supply the options twice, color output escapes will also be 235turned off. 236You may also wish to use the 237.Fl y 238option for scripting dsynth. 239.It Fl N 240Normally 241.Nm 242nices its sub-processes to +10. 243This option disables the feature. 244.El 245.Sh DIRECTIVES 246Generally 247.Nm 248is run with a directive and some directives allow a list of ports to be 249specified. 250This list should be space-delimited in DIR/SUBDIR format, for example: 251.Ar www/chromium . 252For directives with an optional ports list, your current installed set 253of ports will be used if you do not specify a list. You may also 254specify a filename instead of a port to have dsynth read the ports list 255from a text file. Ports are specified by dports directory and subdirectory. 256For example "www/chromium". 257.Bl -tag -width indent 258.It Cm init 259Creates and initializes the 260.Pa /etc/dsynth 261directory if it does not exist. 262This directive will complain and exit if either 263.Pa /etc/dsynth 264or 265.Pa /usr/local/etc/dsynth 266exists. 267It will not create 268.Pa /etc/dsynth 269in this situation. 270.It Cm status 271This will do a dry-run of 272.Cm upgrade-system 273but not actually build anything. 274.It Cm cleanup 275This will clean up any left-over mounts from prior builds. 276.Nm 277attempts to clean up all processes and mounts when you interrupt 278a build but doesn't always succeed. 279.It Cm configure 280NOT CURRENTLY IMPLEMENTED 281.It Cm fetch-only Op Ar ports/everything 282Fetch all source distributions required to build 283the specified target. Specifying 'everything' fetches 284all source distributions required to build the whole 285of dports. 286.Pp 287Any existing distfiles which do not match the expected 288signature will be re-fetched. 289.It Cm upgrade-system 290NOT CURRENTLY IMPLEMENTED. 291Incrementally build and upgrade your locally 292installed packages, then upgrade your local system with them. 293.It Cm list-system 294Write a build list to the file "build.txt". Do not build anything. 295This is typically used on your target system to generate a list for 296dsynth to use as a build list on another system. 297.It Cm prepare-system 298Incrementally build and upgrade your locally installed packages, but 299do not upgrade your system with them. 300.It Cm rebuild-repository 301Build or rebuild the database files for the configured repository. 302.It Cm purge-distfiles 303Delete any obsolete source distribution files. 304.It Cm reset-db 305Delete ports_crc.db from the build directory. 306This database is used to detect changes made to the dports tree. 307It will be regenerated on your next build without forcing any packages to be rebuilt. 308.It Cm status-everything 309This will do a dry-run of a full bulk build of everything, 310but not actually build anything. 311.It Cm everything 312This will build the entire dports tree and then rebuild the repository 313when it finishes. 314.It Cm version 315This is for synth compatibility. 316The version of 317.Nm 318will be printed and the program will exit. 319.It Cm help 320Output a synopsis of options and directives and exit. 321.It Cm status Op Ar ports 322Do a dry-run with 'build' of the given list. 323.It Cm add Op Ar ports 324This directive may be used when the user wishes to add additional 325ports to an existing dsynth run without interrupting and restarting 326the dsynth. 327It can be useful when the user intends to leave dsynth unattended for 328a long period of time and does not wish to interrupt potentially very 329long builds that are already in progress. 330.Pp 331When dsynth completes the current run it will re-exec itself with 332the same primary directive along with all ports specified by any 333.Cm add 334directives made in the interim. 335.Nm 336will still rebuild the repository after the initial run if it would 337normally have done so, but if so it will do it without asking first. 338Only the last rebuild request will potentially be interactive. 339.Pp 340Note that interrupting or killing the running dsynth cleans out any 341ports that might have been added while it was running. This directive 342also has numerous exit/exec lock-file races and is intended to only be used 343manually by the user. 344.It Cm build Op Ar ports 345Incrementally build dports based on the given list. 346When done, ask whether the repository should be rebuilt or not. 347.It Cm just-build Op Ar ports 348Incrementally build dports based on the given list, then 349exits. 350No post-build steps will be taken. 351.It Cm install Op Ar ports 352NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 353list (or using currently installed packages), then rebuild 354the repository and upgrade the system without asking any further 355questions. 356.It Cm force Op Ar ports 357This is the same as 'build' but will delete existing packages first. 358Dependencies are not deleted unless they are out of date. 359.It Cm test Op Ar ports 360This is the same as 'build' but sets the environment variable 361.Ev DEVELOPER 362to 363.Sq yes 364and pre-deletes specified packages. 365Dependencies are not deleted unless they are out of date. 366.It Cm debug Op Ar ports 367This is the same as 'build' but leaves the chroot mounts intact 368upon completion. 369.It Cm monitor Op Ar datfile 370Monitors a running dsynth instance. 371.El 372.Sh PER-PORT OPTIONS 373The 374.Va Directory_options 375configuration variable in 376.Pa /etc/dsynth/dsynth.ini 377points to the configured options directory tree. 378In the base system dports this would be 379.Pa /var/db/ports , 380but you can supply an independent set of ports options for your dsynth 381build if you like. 382The format of the structure in this directory is best described simply by 383CD'ing into a dport, say www/chromium, typing 'make config', and it will 384create a sub-directory and write out a file called 385.Pa /var/db/ports/www_chromium/options . 386.Pp 387For 388.Nm 389you can either point your configuration variable to the system default, 390or you can point it at a dsynth-specific directory and copy the options 391to or construct the options in your dsynth-specific directory tree. 392.Sh HOOKS 393.Nm 394provides several hooks that trigger at specific stages during the 395package building process. 396.Pp 397At the moment hooks are not configurable so the exact executable file is 398expected in the configuration directory with one of the names in the 399list below. 400Hooks are run via 401.Xr execve 2 . 402.Bl -tag -width indent 403.It Pa hook_run_start 404This hook triggers when the overall build process starts. 405.It Pa hook_run_end 406This hook is called when the overall build process ends. 407.It Pa hook_pkg_success 408For each successful port built this hook will trigger. 409.It Pa hook_pkg_failure 410This hook will trigger for each port that fails to build. 411.It Pa hook_pkg_ignored 412Each port that is marked as ignored will make this hook to trigger. 413.It Pa hook_pkg_skipped 414Each skipped port will trigger this hook. 415.El 416.Pp 417A number of environment variables are available for hooks, always in the context 418of an ongoing build and within a specific configuration profile, unless 419overridden from the command-line. 420Some are only available for a specific hook. 421.Bl -tag -width DIR_REPOSITORY 422.It Ev PROFILE 423The configuration profile. 424.It Ev DIR_PACKAGES 425The packages base directory, i.e where index files are generated. 426.It Ev DIR_REPOSITORY 427The packages repository, where the actual package files are stored. 428.It Ev DIR_PORTS 429The ports directory. 430.It Ev DIR_OPTIONS 431The options directory. 432.It Ev DIR_DISTFILES 433The distfiles directory, where the distribution files are stored. 434.It Ev DIR_LOGS 435The logs directory, which is also where the html Report is generated. 436.It Ev DIR_BUILDBASE 437The build base directory. 438.It Ev PORTS_QUEUED 439The number of ports queued to be built (only for 440.Pa hook_run_start ) . 441.It Ev PORTS_BUILT 442The number of successfully built ports (only for 443.Pa hook_run_end ) . 444.It Ev PORTS_FAILED 445The number of ports for which the build failed (only for 446.Pa hook_run_end ) . 447.It Ev PORTS_IGNORED 448The number of ports that where ignored and, hence, not built 449(only for 450.Pa hook_run_end ) . 451.It Ev PORTS_SKIPPED 452The number of ports that were skipped in the build (only for 453.Pa hook_run_end ) . 454.It Ev RESULT 455The result (success, failure, ignored, skipped) for the build of an individual 456port (only for 457.Pa hook_pkg_* ) . 458.It Ev ORIGIN 459The origin of a port (only for 460.Pa hook_pkg_* ) . 461.It Ev FLAVOR 462The flavor of a port (only for 463.Pa hook_pkg_* ) . 464.It Ev PKGNAME 465The port name (only for 466.Pa hook_pkg_* ) . 467.El 468.Sh MISC 469.Pp 470The default setting for 471.Va Meta_version 472is now 2. You can override it with this configuration variable. 473.Pp 474The default setting for 475.Va Check_plist 476is false. You can override it with the 477.Fl P 478option or by setting this configuration variable to true. 479.Sh FILES 480.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 481.It Pa /etc/dsynth/dsynth.ini 482The primary configuration file. 483If not found, 484.Nm 485will also look in 486.Pa /usr/local/etc/dsynth/dsynth.ini . 487.Pp 488.It Pa /etc/dsynth/LiveSystem-make.conf 489Typically contains the environment variables that will be set in 490the workers. 491.Nm 492firewalls the environment it is run under from the environment it 493provides to the workers. 494.Pp 495.It Pa /build/synth/build 496Recommended setting for 497.Va Directory_buildbase , 498contains the build infrastructure... typically a template, mirrored 499system directories, and mount points for all the worker slots. 500The template will be [re]generated if 'pkg' needs to be built or 501if the 502.Pa .template.good 503file in this directory is deleted. 504.Pp 505.It Pa /build/synth/distfiles 506Recommended setting for 507.Va Directory_distfiles , 508ports to a directory into which 509.Nm 510will download any source distribution files required for building. 511.Pp 512.It Pa /build/synth/dports 513Recommended setting for 514.Va Directory_portsdir , 515points to a checked out dports repo. 516Note that 517.Nm 518does not automatically 'git pull' or otherwise synchronize the dports repo, 519you must do that yourself prior to starting a build. 520.Pp 521.It Pa /build/synth/live_packages 522Recommended setting for 523.Va Directory_packages , 524points to a directory which will contain the completed application 525packages. 526.Pp 527.It Pa /build/synth/logs 528Recommended setting for 529.Va Directory_logs , 530all log files will be placed in this directory. 531Special management logfiles begin with the numeral '0' for easily 532location. 533The logfiles for ports while and after building are stored in the 534form subdir____portname.log, with three underscores. 535.Pp 536.It Pa /build/synth/options 537Recommended setting for 538.Va Directory_options , 539where options overrides for specific ports may be located. 540Then either null-mount the system /var/db/ports to this location, or 541construct your own dsynth-specific options. 542See the PER-PORT OPTIONS section above for more information. 543.Pp 544.It Pa / 545Recommended setting for 546.Va Directory_system , 547which 548.Nm 549uses as a basis for creating the jails or chroots in each worker slot 550during building. 551No part of the system root is ever NULL-mounted read-write... it is always 552NULL-mounted read-only. 553Some elements from the system base will be mirrored in the build-base 554as an optimization. 555.Pp 556Note that the packages directory and the distfiles directory is mounted 557read-write in jails or chroots. 558All other r/w filesystems in the workers are 559.Xr tmpfs 5 560based filesystems and will be created and torn-down for each port. 561.Pp 562.It Pa .txz 563.It Pa .tgz 564.It Pa .tar 565.It Pa .tbz 566.It Pa .tzst 567The recommended setting for 568.Va Package_suffix 569is either 570.Pa .txz 571or 572.Pa .tgz . 573Use 574.Pa .txz 575for better compression at the cost of somewhat slower bulk builds due 576to the time overhead for compression and decompression, or 577use 578.Pa .tgz 579for modest compression and very fast compression and decompression. 580Due to the way the builder works, package dependencies are fresly 581installed into the chroot slot for each package being built, so 582decompression time matters. 583.Pp 584.El 585.Sh EXIT STATUS 586.Ex -std 587.Sh SEE ALSO 588.Xr synth 1 , 589.Xr dports 7 590.Sh HISTORY 591The 592.Nm 593utility first appeared in 594.Dx 5.7 . 595.Sh AUTHORS 596.An Matthew Dillon Aq Mt dillon@backplane.com 597