1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dfhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91; Meta_version= 2 (this is the default) 92; Check_plist= false (this is the default) 93; Numa_setsize= 0 (this is the default. 0, 2, or 4) 94Number_of_builders= 8 95Max_jobs_per_builder= 8 96Display_with_ncurses= true 97.Ed 98.Pp 99This places all major directories under 100.Pa /build/synth . 101If you want to use the same dports and the same distfiles as your base 102system, you can null-mount /usr/distfiles onto /build/synth/distfiles 103and /usr/dports onto /build/synth/dports with 104.Pa /etc/fstab 105entries as follows: 106.Bd -literal 107# Device Mountpoint FStype Options DumpPass# 108/usr/distfiles /build/synth/distfiles null rw 4 4 109/usr/dports /build/synth/dports null rw 4 4 110.Ed 111.Pp 112Please set the number of builders and the maximum number of jobs per 113builder according to available system resources. 114Remember that the total 115load on the system can be as high as (builders x jobs), and at least 4x 116that value in processes. 117Systems are typically restricted by memory and CPU horsepower. 118Start conservative and ramp up according to what your system can handle. 119A good rule of thumb is to set workers to the number of CPU threads your 120machine has or to 1/2 the number of gigabytes of memory your system has, 121whichever is lower. 122Then set the jobs per worker to no more than the 123number of CPU threads your machine has. 124.Pp 125.Nm 126has numerous features to manage machine load and swap usage to 127prevent a machine from being overloaded, allowing more workers 128to be configured than you might otherwise think is reasonable 129(which helps a lot when building the smaller ports). 130However, users running this program should be aware that very high loads 131and modest swap use are still likely to develop when building a large 132number of ports or when building very large ports like chromium. 133If the system is not dedicated to building packages you can reduce the 134impact to the rest of the system by running 135.Nm 136at nice +20 and also by reducing the number of workers and number of 137jobs per worker somewhat. 138.Pp 139We recommend that a minimum of 64GB of SSD-based swap be configured, 140or twice as much swap as main memory, whichever is the higher value. 141.Pp 142We recommend a minimum of 500GB of storage be configured in 143.Pa /build 144or wherever you have configured various directories. 145A full set of distfiles requires at least 120GB, a full dports including 146the git repo requires at least 1.5GB, and a full set of built packages 147requires at least 75GB. 148If using a filesystem such as HAMMER or HAMMER2 149which frees space overnight, double all of those numbers. 150.Pp 151The actual build infrastructure uses tmpfs... memory and swap, and does 152not use regular filesystem space. 153.Sh OPTIONS 154.Bl -tag -width indent 155.It Fl d[d...] 156Run in debug mode. 157If specified two or more times this will turn off 158ncurses and output the primary log (00_last_results.log) to the standard 159output, along with additional spew. 160.It Fl f 161Force dangerous directives to run despite pkglist scan errors. Applies only 162to purge-distfiles at the moment. 163.It Fl h 164Quickly output a synopsis of options and directives and exit. 165.It Fl m Ar gb 166Override the default package dependency memory target, in gigabytes. 167The default is 1/2 physical memory. 168The number of workers will be limited 169such that the aggregate size of package dependencies installed in each 170worker slot does not exceed this value. 171.Pp 172This handles a well-known effect where the sheer amount of data that has 173to be installed in tmpfs filesystems for large ports, when multiplied by 174the number of worker slots, can force excessive paging to occur and leave 175preciously little memory available to actually run compiles. 176Some paging 177is necessary to maintain maximum CPU utilization, but excessive paging 178can cause the whole machine to essentially become idle for extended 179periods of time. 180.It Fl M Ar scale 181Override dynamic workers calculation by a specific scale factor. 182Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 18320%, and 1.2 will increase the number of jobs by 20%. And so forth. 184.Pp 185This option may be specified in the range 0.01 to 99.0. Out of range values 186will simply be clipped. 187.It Fl p Ar profile 188Override the global profile default in 189.Pa /etc/dsynth/dsynth.ini , 190allowing you to trivially run whatever profile you like without having to 191edit the configuration file when switching. 192In addition, you can now run any number of dsynth's concurrently on the same 193machine without having to use a jail, each with a different profile, 194as long as the packages, repository, buildbase, and logs directories 195are different. 196.Pp 197Note that the distfiles directory can be shared and will not conflict 198or get confused with concurrent fetches. 199.It Fl s Ar n 200.Nm 201usually slow-starts the worker slots, beginning with one slot and increasing 202by one every 5 seconds until the maximum configured number of workers is 203reached. 204This gives 205.Nm 206a slower ramp that it can load manage against. 207Specifying 0 disables the slow-start feature and the maximum number of 208worker slots (limited by the dependency graph) will be loaded immediately. 209.It Fl v 210Quickly output the version and exit. 211.It Fl x 212.It Fl xx 213Normally dsynth builds a package for any of three reasons: (1) If the contents 214of the ports directory changes, (2) If anything the port depends on requires 215rebuilding so to will the port be rebuilt, (3) If there is no binary package 216already built for the port. 217.Pp 218If this option is specified, the first test is ignored. 219If this option is specified twice, the first and second tests are ignored. 220.It Fl y 221Automatically answer 'y'es to any questions. 222.It Fl D 223Turn on DEVELOPER mode when building ports. 224.It Fl P 225Include the check-plist stage. 226This is the default for the 227.Cm everything 228and 229.Cm test 230directives. 231This feature may also be turned on via the 232.Va Check_plist 233option in the configuration file. 234.It Fl S[S] 235Turn off curses for script friendliness. 236The output will be log 00 and 237should be redirected to /dev/null or something similar. 238If you supply the options twice, color output escapes will also be 239turned off. 240You may also wish to use the 241.Fl y 242option for scripting dsynth. 243.It Fl N 244Normally 245.Nm 246nices its sub-processes to +10. 247This option disables the feature. 248.El 249.Sh DIRECTIVES 250Generally 251.Nm 252is run with a directive and some directives allow a list of ports to be 253specified. 254This list should be space-delimited in DIR/SUBDIR format, for example: 255.Ar www/chromium . 256For directives with an optional ports list, your current installed set 257of ports will be used if you do not specify a list. You may also 258specify a filename instead of a port to have dsynth read the ports list 259from a text file. Ports are specified by dports directory and subdirectory. 260For example "www/chromium". 261.Bl -tag -width indent 262.It Cm init 263Creates and initializes the 264.Pa /etc/dsynth 265directory if it does not exist. 266This directive will complain and exit if either 267.Pa /etc/dsynth 268or 269.Pa /usr/local/etc/dsynth 270exists. 271It will not create 272.Pa /etc/dsynth 273in this situation. 274.It Cm status 275This will do a dry-run of 276.Cm upgrade-system 277but not actually build anything. 278.It Cm cleanup 279This will clean up any left-over mounts from prior builds. 280.Nm 281attempts to clean up all processes and mounts when you interrupt 282a build but doesn't always succeed. 283.It Cm configure 284NOT CURRENTLY IMPLEMENTED 285.It Cm fetch-only Op Ar ports/everything 286Fetch all source distributions required to build 287the specified target. Specifying 'everything' fetches 288all source distributions required to build the whole 289of dports. 290.Pp 291Any existing distfiles which do not match the expected 292signature will be re-fetched. 293.It Cm upgrade-system 294NOT CURRENTLY IMPLEMENTED. 295Incrementally build and upgrade your locally 296installed packages, then upgrade your local system with them. 297.It Cm list-system 298Write a build list to the file "build.txt". Do not build anything. 299This is typically used on your target system to generate a list for 300dsynth to use as a build list on another system. 301.It Cm prepare-system 302Incrementally build and upgrade your locally installed packages, but 303do not upgrade your system with them. 304.It Cm rebuild-repository 305Build or rebuild the database files for the configured repository. 306.It Cm purge-distfiles 307Delete any obsolete source distribution files. This command fails if 308the pkglist scan has errors due to corrupt Makefile's or missing 309dependencies in the dports/ tree. 310The 311.Fl f 312option may be used to force the purge to occur anyway, with the provisio 313that it may end up removing distfiles that you actually need and force 314them to be refetched later. 315.It Cm reset-db 316Delete ports_crc.db from the build directory. 317This database is used to detect changes made to the dports tree. 318It will be regenerated on your next build without forcing any packages to be rebuilt. 319.It Cm status-everything 320This will do a dry-run of a full bulk build of everything, 321but not actually build anything. 322.It Cm everything 323This will build the entire dports tree and then rebuild the repository 324when it finishes. 325.It Cm version 326This is for synth compatibility. 327The version of 328.Nm 329will be printed and the program will exit. 330.It Cm help 331Output a synopsis of options and directives and exit. 332.It Cm status Op Ar ports 333Do a dry-run with 'build' of the given list. 334.It Cm add Op Ar ports 335This directive may be used when the user wishes to add additional 336ports to an existing dsynth run without interrupting and restarting 337the dsynth. 338It can be useful when the user intends to leave dsynth unattended for 339a long period of time and does not wish to interrupt potentially very 340long builds that are already in progress. 341.Pp 342When dsynth completes the current run it will re-exec itself with 343the same primary directive along with all ports specified by any 344.Cm add 345directives made in the interim. 346.Nm 347will still rebuild the repository after the initial run if it would 348normally have done so, but if so it will do it without asking first. 349Only the last rebuild request will potentially be interactive. 350.Pp 351Note that interrupting or killing the running dsynth cleans out any 352ports that might have been added while it was running. This directive 353also has numerous exit/exec lock-file races and is intended to only be used 354manually by the user. 355.It Cm build Op Ar ports 356Incrementally build dports based on the given list. 357When done, ask whether the repository should be rebuilt or not. 358.It Cm just-build Op Ar ports 359Incrementally build dports based on the given list, then 360exits. 361No post-build steps will be taken. 362.It Cm install Op Ar ports 363NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 364list (or using currently installed packages), then rebuild 365the repository and upgrade the system without asking any further 366questions. 367.It Cm force Op Ar ports 368This is the same as 'build' but will delete existing packages first. 369Dependencies are not deleted unless they are out of date. 370.It Cm test Op Ar ports 371This is the same as 'build' but sets the environment variable 372.Ev DEVELOPER 373to 374.Sq yes 375and pre-deletes specified packages. 376Dependencies are not deleted unless they are out of date. 377.It Cm debug Op Ar ports 378This is the same as 'build' but leaves the chroot mounts intact 379upon completion. 380.It Cm monitor Op Ar datfile 381Monitors a running dsynth instance. 382.El 383.Sh PER-PORT OPTIONS 384The 385.Va Directory_options 386configuration variable in 387.Pa /etc/dsynth/dsynth.ini 388points to the configured options directory tree. 389In the base system dports this would be 390.Pa /var/db/ports , 391but you can supply an independent set of ports options for your dsynth 392build if you like. 393The format of the structure in this directory is best described simply by 394CD'ing into a dport, say www/chromium, typing 'make config', and it will 395create a sub-directory and write out a file called 396.Pa /var/db/ports/www_chromium/options . 397.Pp 398For 399.Nm 400you can either point your configuration variable to the system default, 401or you can point it at a dsynth-specific directory and copy the options 402to or construct the options in your dsynth-specific directory tree. 403.Sh HOOKS 404.Nm 405provides several hooks that trigger at specific stages during the 406package building process. 407.Pp 408At the moment hooks are not configurable so the exact executable file is 409expected in the configuration directory with one of the names in the 410list below. 411Hooks are run via 412.Xr execve 2 . 413.Bl -tag -width indent 414.It Pa hook_run_start 415This hook triggers when the overall build process starts. 416.It Pa hook_run_end 417This hook is called when the overall build process ends. 418.It Pa hook_pkg_success 419For each successful port built this hook will trigger. 420.It Pa hook_pkg_failure 421This hook will trigger for each port that fails to build. 422.It Pa hook_pkg_ignored 423Each port that is marked as ignored will make this hook to trigger. 424.It Pa hook_pkg_skipped 425Each skipped port will trigger this hook. 426.El 427.Pp 428A number of environment variables are available for hooks, always in the context 429of an ongoing build and within a specific configuration profile, unless 430overridden from the command-line. 431Some are only available for a specific hook. 432.Bl -tag -width DIR_REPOSITORY 433.It Ev PROFILE 434The configuration profile. 435.It Ev DIR_PACKAGES 436The packages base directory, i.e where index files are generated. 437.It Ev DIR_REPOSITORY 438The packages repository, where the actual package files are stored. 439.It Ev DIR_PORTS 440The ports directory. 441.It Ev DIR_OPTIONS 442The options directory. 443.It Ev DIR_DISTFILES 444The distfiles directory, where the distribution files are stored. 445.It Ev DIR_LOGS 446The logs directory, which is also where the html Report is generated. 447.It Ev DIR_BUILDBASE 448The build base directory. 449.It Ev PORTS_QUEUED 450The number of ports queued to be built (only for 451.Pa hook_run_start ) . 452.It Ev PORTS_BUILT 453The number of successfully built ports (only for 454.Pa hook_run_end ) . 455.It Ev PORTS_FAILED 456The number of ports for which the build failed (only for 457.Pa hook_run_end ) . 458.It Ev PORTS_IGNORED 459The number of ports that where ignored and, hence, not built 460(only for 461.Pa hook_run_end ) . 462.It Ev PORTS_SKIPPED 463The number of ports that were skipped in the build (only for 464.Pa hook_run_end ) . 465.It Ev RESULT 466The result (success, failure, ignored, skipped) for the build of an individual 467port (only for 468.Pa hook_pkg_* ) . 469.It Ev ORIGIN 470The origin of a port (only for 471.Pa hook_pkg_* ) . 472.It Ev FLAVOR 473The flavor of a port (only for 474.Pa hook_pkg_* ) . 475.It Ev PKGNAME 476The port name (only for 477.Pa hook_pkg_* ) . 478.El 479.Sh MISC 480.Pp 481The default setting for 482.Va Meta_version 483is now 2. You can override it with this configuration variable. 484.Pp 485The default setting for 486.Va Check_plist 487is false. You can override it with the 488.Fl P 489option or by setting this configuration variable to true. 490.Pp 491The default setting for 492.Va Numa_setsize 493is 0, disabling any NUMA related CPU partitioning. This is the recommended 494setting as it allows the scheduler the freedom to maximize CPU utilization, 495particularly when doing incremental bulks. The only other values that have 496a decent chance of improving full bulk performance are 2 or 4. Any higher 497and CPU utilization will suffer from periods of high idle. 498.Sh FILES 499.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 500.It Pa /etc/dsynth/dsynth.ini 501The primary configuration file. 502If not found, 503.Nm 504will also look in 505.Pa /usr/local/etc/dsynth/dsynth.ini . 506.Pp 507.It Pa /etc/dsynth/LiveSystem-make.conf 508Typically contains the environment variables that will be set in 509the workers. 510.Nm 511firewalls the environment it is run under from the environment it 512provides to the workers. 513.Pp 514.It Pa /build/synth/build 515Recommended setting for 516.Va Directory_buildbase , 517contains the build infrastructure... typically a template, mirrored 518system directories, and mount points for all the worker slots. 519The template will be [re]generated if 'pkg' needs to be built or 520if the 521.Pa .template.good 522file in this directory is deleted. 523.Pp 524.It Pa /build/synth/distfiles 525Recommended setting for 526.Va Directory_distfiles , 527ports to a directory into which 528.Nm 529will download any source distribution files required for building. 530.Pp 531.It Pa /build/synth/dports 532Recommended setting for 533.Va Directory_portsdir , 534points to a checked out dports repo. 535Note that 536.Nm 537does not automatically 'git pull' or otherwise synchronize the dports repo, 538you must do that yourself prior to starting a build. 539.Pp 540.It Pa /build/synth/live_packages 541Recommended setting for 542.Va Directory_packages , 543points to a directory which will contain the completed application 544packages. 545.Pp 546.It Pa /build/synth/logs 547Recommended setting for 548.Va Directory_logs , 549all log files will be placed in this directory. 550Special management logfiles begin with the numeral '0' for easily 551location. 552The logfiles for ports while and after building are stored in the 553form subdir____portname.log, with three underscores. 554.Pp 555.It Pa /build/synth/options 556Recommended setting for 557.Va Directory_options , 558where options overrides for specific ports may be located. 559Then either null-mount the system /var/db/ports to this location, or 560construct your own dsynth-specific options. 561See the PER-PORT OPTIONS section above for more information. 562.Pp 563.It Pa / 564Recommended setting for 565.Va Directory_system , 566which 567.Nm 568uses as a basis for creating the jails or chroots in each worker slot 569during building. 570No part of the system root is ever NULL-mounted read-write... it is always 571NULL-mounted read-only. 572Some elements from the system base will be mirrored in the build-base 573as an optimization. 574.Pp 575Note that the packages directory and the distfiles directory is mounted 576read-write in jails or chroots. 577All other r/w filesystems in the workers are 578.Xr tmpfs 5 579based filesystems and will be created and torn-down for each port. 580.Pp 581.It Pa .txz 582.It Pa .tgz 583.It Pa .tar 584.It Pa .tbz 585.It Pa .tzst 586The recommended setting for 587.Va Package_suffix 588is either 589.Pa .txz 590or 591.Pa .tgz . 592Use 593.Pa .txz 594for better compression at the cost of somewhat slower bulk builds due 595to the time overhead for compression and decompression, or 596use 597.Pa .tgz 598for modest compression and very fast compression and decompression. 599Due to the way the builder works, package dependencies are fresly 600installed into the chroot slot for each package being built, so 601decompression time matters. 602.Pp 603.El 604.Sh EXIT STATUS 605.Ex -std 606.Sh SEE ALSO 607.Xr synth 1 , 608.Xr dports 7 609.Sh HISTORY 610The 611.Nm 612utility first appeared in 613.Dx 5.7 . 614.Sh AUTHORS 615.An Matthew Dillon Aq Mt dillon@backplane.com 616