1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd January 26, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDSN 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Ar directive 48.Op origins 49.Nm 50.Ar help 51.Sh DESCRIPTION 52The 53.Nm 54utility allows a user to build and maintain part or all of dports 55locally. 56.Nm 57figures out the dependency topology of the dport(s) for you and 58is capable of building any number of ports concurrently based 59on the configuration parameters you supply. 60.Pp 61also detects changes made to the ports tree and rebuilds packages 62and anything that depends on those packages as needed. 63.Pp 64.Nm 65is based on an application called 66.Xr synth 1 67which was written by John Marino in Ada and served as the conceptual base 68for this program. 69.Nm 70is written in C and designed to be as portable as possible given a 71ports-style infrastructure. 72.Pp 73Our recommended build topology is with a configuration as follows: 74.Bd -literal 75[Global Configuration] 76profile_selected= LiveSystem 77 78[LiveSystem] 79Operating_system= DragonFly 80Directory_packages= /build/synth/live_packages 81Directory_repository= /build/synth/live_packages/All 82Directory_portsdir= /build/synth/dports 83Directory_options= /build/synth/options 84Directory_distfiles= /build/synth/distfiles 85Directory_buildbase= /build/synth/build 86Directory_logs= /build/synth/logs 87Directory_ccache= disabled 88Directory_system= / 89Package_suffix= .txz 90Number_of_builders= 8 91Max_jobs_per_builder= 8 92Display_with_ncurses= true 93.Ed 94.Pp 95This places all major directories under 96.Pa /build/synth . 97If you want to use the same dports and the same distfiles as your base 98system, you can null-mount /usr/distfiles onto /build/synth/distfiles 99and /usr/dports onto /build/synth/dports with 100.Pa /etc/fstab 101entries as follows: 102.Bd -literal 103# Device Mountpoint FStype Options DumpPass# 104/usr/distfiles /build/synth/distfiles null rw 4 4 105/usr/dports /build/synth/dports null rw 4 4 106.Ed 107.Pp 108Please set the number of builders and the maximum number of jobs per 109builder according to available system resources. 110Remember that the total 111load on the system can be as high as (builders x jobs), and at least 4x 112that value in processes. 113Systems are typically restricted by memory and CPU horsepower. 114Start conservative and ramp up according to what your system can handle. 115A good rule of thumb is to set workers to the number of CPU threads your 116machine has or to 1/2 the number of gigabytes of memory your system has, 117whichever is lower. 118Then set the jobs per worker to no more than the 119number of CPU threads your machine has. 120.Pp 121.Nm 122has numerous features to manage machine load and swap usage to 123prevent a machine from being overloaded, allowing more workers 124to be configured than you might otherwise think is reasonable 125(which helps a lot when building the smaller ports). 126However, users running this program should be aware that very high loads 127and modest swap use are still likely to develop when building a large 128number of ports or when building very large ports like chromium. 129If the system is not dedicated to building packages you can reduce the 130impact to the rest of the system by running 131.Nm 132at nice +20 and also by reducing the number of workers and number of 133jobs per worker somewhat. 134.Pp 135We recommend that a minimum of 64GB of SSD-based swap be configured, 136or twice as much swap as main memory, whichever is the higher value. 137.Pp 138We recommend a minimum of 500GB of storage be configured in 139.Pa /build 140or wherever you have configured various directories. 141A full set of distfiles requires at least 120GB, a full dports including 142the git repo requires at least 1.5GB, and a full set of built packages 143requires at least 75GB. 144If using a filesystem such as HAMMER or HAMMER2 145which frees space overnight, double all of those numbers. 146.Pp 147The actual build infrastructure uses tmpfs... memory and swap, and does 148not use regular filesystem space. 149.Sh OPTIONS 150.Bl -tag -width indent 151.It Fl d[d...] 152Run in debug mode. 153If specified two or more times this will turn off 154ncurses and output the primary log (00_last_results.log) to the standard 155output, along with additional spew. 156.It Fl h 157Quickly output a synopsis of options and directives and exit. 158.It Fl m Ar gb 159Override the default package dependency memory target, in gigabytes. 160The default is 1/2 physical memory. 161The number of workers will be limited 162such that the aggregate size of package dependencies installed in each 163worker slot does not exceed this value. 164.Pp 165This handles a well-known effect where the sheer amount of data that has 166to be installed in tmpfs filesystems for large ports, when multiplied by 167the number of worker slots, can force excessive paging to occur and leave 168preciously little memory available to actually run compiles. 169Some paging 170is necessary to maintain maximum CPU utilization, but excessive paging 171can cause the whole machine to essentially become idle for extended 172periods of time. 173.It Fl p Ar profile 174Override the global profile default in 175.Pa /etc/dsynth/dsynth.ini , 176allowing you to trivially run whatever profile you like without having to 177edit the configuration file when switching. 178In addition, you can now run any number of dsynth's concurrently on the same 179machine without having to use a jail, each with a different profile, 180as long as the packages, repository, buildbase, and logs directories 181are different. 182.Pp 183Note that the distfiles directory can be shared and will not conflict 184or get confused with concurrent fetches. 185.It Fl s Ar n 186.Nm 187usually slow-starts the worker slots, beginning with one slot and increasing 188by one every 5 seconds until the maximum configured number of workers is 189reached. 190This gives 191.Nm 192a slower ramp that it can load manage against. 193Specifying 0 disables the slow-start feature and the maximum number of 194worker slots (limited by the dependency graph) will be loaded immediately. 195.It Fl v 196Quickly output the version and exit. 197.It Fl x 198.It Fl xx 199Normally dsynth builds a package for any of three reasons: (1) If the contents 200of the ports directory changes, (2) If anything the port depends on requires 201rebuilding so to will the port be rebuilt, (3) If there is no binary package 202already built for the port. 203.Pp 204If this option is specified, the first test is ignored. 205If this option is specified twice, the first and second tests are ignored. 206.It Fl y 207Automatically answer 'y'es to any questions. 208.It Fl D 209Turn on DEVELOPER mode when building ports. 210.It Fl P 211Include the check-plist stage. 212This is the default for the 213.Cm everything 214directive. 215.It Fl S[S] 216Turn off curses for script friendliness. 217The output will be log 00 and 218should be redirected to /dev/null or something similar. 219If you supply the options twice, color output escapes will also be 220turned off. 221You may also wish to use the 222.Fl y 223option for scripting dsynth. 224.It Fl N 225Normally 226.Nm 227nices its sub-processes to +10. 228This option disables the feature. 229.El 230.Sh DIRECTIVES 231Generally 232.Nm 233is run with a directive and some directives allow a list of ports to be 234specified. 235This list should be space-delimited in DIR/SUBDIR format, for example: 236.Ar www/chromium . 237For directives with an optional ports list, your current installed set 238of ports will be used if you do not specify a list. 239.Bl -tag -width indent 240.It Cm init 241Creates and initializes the 242.Pa /etc/dsynth 243directory if it does not exst. 244This directive will complain and exit if either 245.Pa /etc/dsynth 246or 247.Pa /usr/local/etc/dsynth 248exists. 249It will not create 250.Pa /etc/dsynth 251in this situation. 252.It Cm status 253This will do a dry-run of 254.Cm upgrade-system 255but not actually build anything. 256.It Cm cleanup 257This will clean up any left-over mounts from prior builds. 258.Nm 259attempts to clean up all processes and mounts when you interrupt 260a build but doesn't always succeed. 261.It Cm configure 262NOT CURRENTLY IMPLEMENTED 263.It Cm upgrade-system 264NOT CURRENTLY IMPLEMENTED. 265Incrementally build and upgrade your locally 266installed packages, then upgrade your local system with them. 267.It Cm prepare-system 268Incrementally build and upgrade your locally installed packages, but 269do not upgrade your system with them. 270.It Cm rebuild-repository 271Build or rebuild the database files for the configured repository. 272.It Cm purge-distfiles 273Delete any obsolete source distribution files. 274.It Cm reset-db 275Delete ports_crc.db from the build directory. 276This database is used to detect changes made to the dports tree. 277It will be regenerated on your next build without forcing any packages to be rebuilt. 278.It Cm status-everything 279This will do a dry-run of a full bulk build of everything, 280but not actually build anything. 281.It Cm everything 282This will build the entire dports tree and then rebuild the repository 283when it finishes. 284.It Cm version 285This is for synth compatibility. 286The version of 287.Nm 288will be printed and the program will exit. 289.It Cm help 290Output a synopsis of options and directives and exit. 291.It Cm status Op Ar ports 292Do a dry-run with 'build' of the given list. 293.It Cm build Op Ar ports 294Incrementally build dports based on the given list. 295When done, ask whether the repository should be rebuilt or not. 296.It Cm just-build Op Ar ports 297Incrementally build dports based on the given list, then 298exits. 299No post-build steps will be taken. 300.It Cm install Op Ar ports 301NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 302list (or using currently installed packages), then rebuild 303the repository and upgrade the system without asking any further 304questions. 305.It Cm force Op Ar ports 306This is the same as 'build' but will delete existing packages first. 307Dependencies are not deleted unless they are out of date. 308.It Cm test Op Ar ports 309This is the same as 'build' but sets the environment variable 310.Ev DEVELOPER 311to 312.Sq yes 313and pre-deletes specified packages. 314Dependencies are not deleted unless they are out of date. 315.It Cm debug Op Ar ports 316This is the same as 'build' but leaves the chroot mounts intact 317upon completion. 318.It Cm monitor Op Ar datfile 319Monitors a running dsynth instance. 320.El 321.Sh FILES 322.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 323.It Pa /etc/dsynth/dsynth.ini 324The primary configuration file. 325If not found, 326.Nm 327will also look in 328.Pa /usr/local/etc/dsynth/dsynth.ini . 329.Pp 330.It Pa /etc/dsynth/LiveSystem-make.conf 331Typically contains the environment variables that will be set in 332the workers. 333.Nm 334firewalls the environment it is run under from the environment it 335provides to the workers. 336.Pp 337.It Pa /build/synth/build 338Recommended setting for 339.Va Directory_buildbase , 340contains the build infrastructure... typically a template, mirrored 341system directories, and mount points for all the worker slots. 342The template will be [re]generated if 'pkg' needs to be built or 343if the 344.Pa .template.good 345file in this directory is deleted. 346.Pp 347.It Pa /build/synth/distfiles 348Recommended setting for 349.Va Directory_distfiles , 350ports to a directory into which 351.Nm 352will download any source distribution files required for building. 353.Pp 354.It Pa /build/synth/dports 355Recommended setting for 356.Va Directory_portsdir , 357points to a checked out dports repo. 358Note that 359.Nm 360does not automatically 'git pull' or otherwise synchronize the dports repo, 361you must do that yourself prior to starting a build. 362.Pp 363.It Pa /build/synth/live_packages 364Recommended setting for 365.Va Directory_packages , 366points to a directory which will contain the completed application 367packages. 368.Pp 369.It Pa /build/synth/logs 370Recommended setting for 371.Va Directory_logs , 372all log files will be placed in this directory. 373Special management logfiles begin with the numeral '0' for easily 374location. 375The logfiles for ports while and after building are stored in the 376form subdir____portname.log, with three underscores. 377.Pp 378.It Pa /build/synth/options 379Recommended setting for 380.Va Directory_options , 381where options overrides for specific ports may be located. 382.Pp 383.It Pa / 384Recommended setting for 385.Va Directory_system , 386which 387.Nm 388uses as a basis for creating the jails or chroots in each worker slot 389during building. 390No part of the system root is ever NULL-mounted read-write... it is always 391NULL-mounted read-only. 392Some elements from the system base will be mirrored in the build-base 393as an optimization. 394.Pp 395Note that the packages directory and the distfiles directory is mounted 396read-write in jails or chroots. 397All other r/w filesystems in the workers are 398.Xr tmpfs 5 399based filesystems and will be created and torn-down for each port. 400.Pp 401.It Pa .txz 402.It Pa .tgz 403.It Pa .tar 404.It Pa .tbz 405.It Pa .tzst 406The recommended setting for 407.Va Package_suffix 408is either 409.Pa .txz 410or 411.Pa .tgz . 412Use 413.Pa .txz 414for better compression at the cost of somewhat slower bulk builds due 415to the time overhead for compression and decompression, or 416use 417.Pa .tgz 418for modest compression and very fast compression and decompression. 419Due to the way the builder works, package dependencies are fresly 420installed into the chroot slot for each package being built, so 421decompression time matters. 422.El 423.Sh EXIT STATUS 424.Ex -std 425.Sh SEE ALSO 426.Xr synth 1 , 427.Xr dports 7 428.Sh HISTORY 429The 430.Nm 431utility first appeared in 432.Dx 5.7 . 433.Sh AUTHORS 434.An Matthew Dillon Aq Mt dillon@backplane.com 435