1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91; Meta_version= 2 (this is the default) 92Number_of_builders= 8 93Max_jobs_per_builder= 8 94Display_with_ncurses= true 95.Ed 96.Pp 97This places all major directories under 98.Pa /build/synth . 99If you want to use the same dports and the same distfiles as your base 100system, you can null-mount /usr/distfiles onto /build/synth/distfiles 101and /usr/dports onto /build/synth/dports with 102.Pa /etc/fstab 103entries as follows: 104.Bd -literal 105# Device Mountpoint FStype Options DumpPass# 106/usr/distfiles /build/synth/distfiles null rw 4 4 107/usr/dports /build/synth/dports null rw 4 4 108.Ed 109.Pp 110Please set the number of builders and the maximum number of jobs per 111builder according to available system resources. 112Remember that the total 113load on the system can be as high as (builders x jobs), and at least 4x 114that value in processes. 115Systems are typically restricted by memory and CPU horsepower. 116Start conservative and ramp up according to what your system can handle. 117A good rule of thumb is to set workers to the number of CPU threads your 118machine has or to 1/2 the number of gigabytes of memory your system has, 119whichever is lower. 120Then set the jobs per worker to no more than the 121number of CPU threads your machine has. 122.Pp 123.Nm 124has numerous features to manage machine load and swap usage to 125prevent a machine from being overloaded, allowing more workers 126to be configured than you might otherwise think is reasonable 127(which helps a lot when building the smaller ports). 128However, users running this program should be aware that very high loads 129and modest swap use are still likely to develop when building a large 130number of ports or when building very large ports like chromium. 131If the system is not dedicated to building packages you can reduce the 132impact to the rest of the system by running 133.Nm 134at nice +20 and also by reducing the number of workers and number of 135jobs per worker somewhat. 136.Pp 137We recommend that a minimum of 64GB of SSD-based swap be configured, 138or twice as much swap as main memory, whichever is the higher value. 139.Pp 140We recommend a minimum of 500GB of storage be configured in 141.Pa /build 142or wherever you have configured various directories. 143A full set of distfiles requires at least 120GB, a full dports including 144the git repo requires at least 1.5GB, and a full set of built packages 145requires at least 75GB. 146If using a filesystem such as HAMMER or HAMMER2 147which frees space overnight, double all of those numbers. 148.Pp 149The actual build infrastructure uses tmpfs... memory and swap, and does 150not use regular filesystem space. 151.Sh OPTIONS 152.Bl -tag -width indent 153.It Fl d[d...] 154Run in debug mode. 155If specified two or more times this will turn off 156ncurses and output the primary log (00_last_results.log) to the standard 157output, along with additional spew. 158.It Fl h 159Quickly output a synopsis of options and directives and exit. 160.It Fl m Ar gb 161Override the default package dependency memory target, in gigabytes. 162The default is 1/2 physical memory. 163The number of workers will be limited 164such that the aggregate size of package dependencies installed in each 165worker slot does not exceed this value. 166.Pp 167This handles a well-known effect where the sheer amount of data that has 168to be installed in tmpfs filesystems for large ports, when multiplied by 169the number of worker slots, can force excessive paging to occur and leave 170preciously little memory available to actually run compiles. 171Some paging 172is necessary to maintain maximum CPU utilization, but excessive paging 173can cause the whole machine to essentially become idle for extended 174periods of time. 175.It Fl M Ar scale 176Override dynamic workers calculation by a specific scale factor. 177Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 17820%, and 1.2 will increase the number of jobs by 20%. And so forth. 179.Pp 180This option may be specified in the range 0.01 to 99.0. Out of range values 181will simply be clipped. 182.It Fl p Ar profile 183Override the global profile default in 184.Pa /etc/dsynth/dsynth.ini , 185allowing you to trivially run whatever profile you like without having to 186edit the configuration file when switching. 187In addition, you can now run any number of dsynth's concurrently on the same 188machine without having to use a jail, each with a different profile, 189as long as the packages, repository, buildbase, and logs directories 190are different. 191.Pp 192Note that the distfiles directory can be shared and will not conflict 193or get confused with concurrent fetches. 194.It Fl s Ar n 195.Nm 196usually slow-starts the worker slots, beginning with one slot and increasing 197by one every 5 seconds until the maximum configured number of workers is 198reached. 199This gives 200.Nm 201a slower ramp that it can load manage against. 202Specifying 0 disables the slow-start feature and the maximum number of 203worker slots (limited by the dependency graph) will be loaded immediately. 204.It Fl v 205Quickly output the version and exit. 206.It Fl x 207.It Fl xx 208Normally dsynth builds a package for any of three reasons: (1) If the contents 209of the ports directory changes, (2) If anything the port depends on requires 210rebuilding so to will the port be rebuilt, (3) If there is no binary package 211already built for the port. 212.Pp 213If this option is specified, the first test is ignored. 214If this option is specified twice, the first and second tests are ignored. 215.It Fl y 216Automatically answer 'y'es to any questions. 217.It Fl D 218Turn on DEVELOPER mode when building ports. 219.It Fl P 220Include the check-plist stage. 221This is the default for the 222.Cm everything 223directive. 224.It Fl S[S] 225Turn off curses for script friendliness. 226The output will be log 00 and 227should be redirected to /dev/null or something similar. 228If you supply the options twice, color output escapes will also be 229turned off. 230You may also wish to use the 231.Fl y 232option for scripting dsynth. 233.It Fl N 234Normally 235.Nm 236nices its sub-processes to +10. 237This option disables the feature. 238.El 239.Sh DIRECTIVES 240Generally 241.Nm 242is run with a directive and some directives allow a list of ports to be 243specified. 244This list should be space-delimited in DIR/SUBDIR format, for example: 245.Ar www/chromium . 246For directives with an optional ports list, your current installed set 247of ports will be used if you do not specify a list. 248.Bl -tag -width indent 249.It Cm init 250Creates and initializes the 251.Pa /etc/dsynth 252directory if it does not exist. 253This directive will complain and exit if either 254.Pa /etc/dsynth 255or 256.Pa /usr/local/etc/dsynth 257exists. 258It will not create 259.Pa /etc/dsynth 260in this situation. 261.It Cm status 262This will do a dry-run of 263.Cm upgrade-system 264but not actually build anything. 265.It Cm cleanup 266This will clean up any left-over mounts from prior builds. 267.Nm 268attempts to clean up all processes and mounts when you interrupt 269a build but doesn't always succeed. 270.It Cm configure 271NOT CURRENTLY IMPLEMENTED 272.It Cm fetch-only Op Ar ports/everything 273Fetch all source distributions required to build 274the specified target. Specifying 'everything' fetches 275all source distributions required to build the whole 276of dports. 277.Pp 278Any existing distfiles which do not match the expected 279signature will be re-fetched. 280.It Cm upgrade-system 281NOT CURRENTLY IMPLEMENTED. 282Incrementally build and upgrade your locally 283installed packages, then upgrade your local system with them. 284.It Cm prepare-system 285Incrementally build and upgrade your locally installed packages, but 286do not upgrade your system with them. 287.It Cm rebuild-repository 288Build or rebuild the database files for the configured repository. 289.It Cm purge-distfiles 290Delete any obsolete source distribution files. 291.It Cm reset-db 292Delete ports_crc.db from the build directory. 293This database is used to detect changes made to the dports tree. 294It will be regenerated on your next build without forcing any packages to be rebuilt. 295.It Cm status-everything 296This will do a dry-run of a full bulk build of everything, 297but not actually build anything. 298.It Cm everything 299This will build the entire dports tree and then rebuild the repository 300when it finishes. 301.It Cm version 302This is for synth compatibility. 303The version of 304.Nm 305will be printed and the program will exit. 306.It Cm help 307Output a synopsis of options and directives and exit. 308.It Cm status Op Ar ports 309Do a dry-run with 'build' of the given list. 310.It Cm build Op Ar ports 311Incrementally build dports based on the given list. 312When done, ask whether the repository should be rebuilt or not. 313.It Cm just-build Op Ar ports 314Incrementally build dports based on the given list, then 315exits. 316No post-build steps will be taken. 317.It Cm install Op Ar ports 318NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 319list (or using currently installed packages), then rebuild 320the repository and upgrade the system without asking any further 321questions. 322.It Cm force Op Ar ports 323This is the same as 'build' but will delete existing packages first. 324Dependencies are not deleted unless they are out of date. 325.It Cm test Op Ar ports 326This is the same as 'build' but sets the environment variable 327.Ev DEVELOPER 328to 329.Sq yes 330and pre-deletes specified packages. 331Dependencies are not deleted unless they are out of date. 332.It Cm debug Op Ar ports 333This is the same as 'build' but leaves the chroot mounts intact 334upon completion. 335.It Cm monitor Op Ar datfile 336Monitors a running dsynth instance. 337.El 338.Sh HOOKS 339.Nm 340provides several hooks that trigger at specific stages during the 341package building process. 342.Pp 343At the moment hooks are not configurable so the exact executable file is 344expected in the configuration directory with one of the names in the 345list below. 346Hooks are run via 347.Xr execve 2 . 348.Bl -tag -width indent 349.It Pa hook_run_start 350This hook triggers when the overall build process starts. 351.It Pa hook_run_end 352This hook is called when the overall build process ends. 353.It Pa hook_pkg_success 354For each successful port built this hook will trigger. 355.It Pa hook_pkg_failure 356This hook will trigger for each port that fails to build. 357.It Pa hook_pkg_ignored 358Each port that is marked as ignored will make this hook to trigger. 359.It Pa hook_pkg_skipped 360Each skipped port will trigger this hook. 361.El 362.Pp 363A number of environment variables are available for hooks, always in the context 364of an ongoing build and within a specific configuration profile, unless 365overridden from the command-line. 366Some are only available for a specific hook. 367.Bl -tag -width DIR_REPOSITORY 368.It Ev PROFILE 369The configuration profile. 370.It Ev DIR_PACKAGES 371The packages base directory, i.e where index files are generated. 372.It Ev DIR_REPOSITORY 373The packages repository, where the actual package files are stored. 374.It Ev DIR_PORTS 375The ports directory. 376.It Ev DIR_OPTIONS 377The options directory. 378.It Ev DIR_DISTFILES 379The distfiles directory, where the distribution files are stored. 380.It Ev DIR_LOGS 381The logs directory, which is also where the html Report is generated. 382.It Ev DIR_BUILDBASE 383The build base directory. 384.It Ev PORTS_QUEUED 385The number of ports queued to be built (only for 386.Pa hook_run_start ) . 387.It Ev PORTS_BUILT 388The number of successfully built ports (only for 389.Pa hook_run_end ) . 390.It Ev PORTS_FAILED 391The number of ports for which the build failed (only for 392.Pa hook_run_end ) . 393.It Ev PORTS_IGNORED 394The number of ports that where ignored and, hence, not built 395(only for 396.Pa hook_run_end ) . 397.It Ev PORTS_SKIPPED 398The number of ports that were skipped in the build (only for 399.Pa hook_run_end ) . 400.It Ev RESULT 401The result (success, failure, ignored, skipped) for the build of an individual 402port (only for 403.Pa hook_pkg_* ) . 404.It Ev ORIGIN 405The origin of a port (only for 406.Pa hook_pkg_* ) . 407.It Ev FLAVOR 408The flavor of a port (only for 409.Pa hook_pkg_* ) . 410.It Ev PKGNAME 411The port name (only for 412.Pa hook_pkg_* ) . 413.El 414.Sh FILES 415.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 416.It Pa /etc/dsynth/dsynth.ini 417The primary configuration file. 418If not found, 419.Nm 420will also look in 421.Pa /usr/local/etc/dsynth/dsynth.ini . 422.Pp 423.It Pa /etc/dsynth/LiveSystem-make.conf 424Typically contains the environment variables that will be set in 425the workers. 426.Nm 427firewalls the environment it is run under from the environment it 428provides to the workers. 429.Pp 430.It Pa /build/synth/build 431Recommended setting for 432.Va Directory_buildbase , 433contains the build infrastructure... typically a template, mirrored 434system directories, and mount points for all the worker slots. 435The template will be [re]generated if 'pkg' needs to be built or 436if the 437.Pa .template.good 438file in this directory is deleted. 439.Pp 440.It Pa /build/synth/distfiles 441Recommended setting for 442.Va Directory_distfiles , 443ports to a directory into which 444.Nm 445will download any source distribution files required for building. 446.Pp 447.It Pa /build/synth/dports 448Recommended setting for 449.Va Directory_portsdir , 450points to a checked out dports repo. 451Note that 452.Nm 453does not automatically 'git pull' or otherwise synchronize the dports repo, 454you must do that yourself prior to starting a build. 455.Pp 456.It Pa /build/synth/live_packages 457Recommended setting for 458.Va Directory_packages , 459points to a directory which will contain the completed application 460packages. 461.Pp 462.It Pa /build/synth/logs 463Recommended setting for 464.Va Directory_logs , 465all log files will be placed in this directory. 466Special management logfiles begin with the numeral '0' for easily 467location. 468The logfiles for ports while and after building are stored in the 469form subdir____portname.log, with three underscores. 470.Pp 471.It Pa /build/synth/options 472Recommended setting for 473.Va Directory_options , 474where options overrides for specific ports may be located. 475.Pp 476.It Pa / 477Recommended setting for 478.Va Directory_system , 479which 480.Nm 481uses as a basis for creating the jails or chroots in each worker slot 482during building. 483No part of the system root is ever NULL-mounted read-write... it is always 484NULL-mounted read-only. 485Some elements from the system base will be mirrored in the build-base 486as an optimization. 487.Pp 488Note that the packages directory and the distfiles directory is mounted 489read-write in jails or chroots. 490All other r/w filesystems in the workers are 491.Xr tmpfs 5 492based filesystems and will be created and torn-down for each port. 493.Pp 494.It Pa .txz 495.It Pa .tgz 496.It Pa .tar 497.It Pa .tbz 498.It Pa .tzst 499The recommended setting for 500.Va Package_suffix 501is either 502.Pa .txz 503or 504.Pa .tgz . 505Use 506.Pa .txz 507for better compression at the cost of somewhat slower bulk builds due 508to the time overhead for compression and decompression, or 509use 510.Pa .tgz 511for modest compression and very fast compression and decompression. 512Due to the way the builder works, package dependencies are fresly 513installed into the chroot slot for each package being built, so 514decompression time matters. 515.Pp 516.It 2 517The default setting for 518.Va Meta_version 519is now 2. You can override it with this configuration variable. 520.El 521.Sh EXIT STATUS 522.Ex -std 523.Sh SEE ALSO 524.Xr synth 1 , 525.Xr dports 7 526.Sh HISTORY 527The 528.Nm 529utility first appeared in 530.Dx 5.7 . 531.Sh AUTHORS 532.An Matthew Dillon Aq Mt dillon@backplane.com 533