1.\" $OpenBSD: mirroring-ports.7,v 1.11 2003/07/31 10:53:50 espie Exp $ 2.\" 3.\" Copyright (c) 2000 Marc Espie 4.\" 5.\" All rights reserved. 6.\" 7.\" Redistribution and use in source and binary forms, with or without 8.\" modification, are permitted provided that the following conditions 9.\" are met: 10.\" 1. Redistributions of source code must retain the above copyright 11.\" notice, this list of conditions and the following disclaimer. 12.\" 2. Redistributions in binary form must reproduce the above copyright 13.\" notice, this list of conditions and the following disclaimer in the 14.\" documentation and/or other materials provided with the distribution. 15.\" 16.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 17.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 18.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 19.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 20.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 21.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 22.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 23.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 25.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26.\" 27.Dd February 19, 2000 28.Dt MIRRORING-PORTS 7 29.Os 30.Sh NAME 31.Nm mirroring-ports 32.Nd how to build a mirror for ports distfiles 33.Sh DESCRIPTION 34The 35.Nm OpenBSD Ports Collection 36offers some powerful tools to mirror software sources. 37The ports infrastructure provides a 38.Ar mirror-maker 39target that can be used to build a Makefile to facilitate mirroring 40distfiles. 41This target builds 42.Ic ${DISTDIR}/Makefile , 43which can be used on almost any Unix-like machine to mirror 44.Ox 45distfiles. 46The variable 47.Ev RECURSIVE_FETCH_LIST 48can be set to 49.Sq Yes 50if port dependencies should also be recorded. 51.Pp 52A sample Makefile entry is formatted like this: 53.Bd -literal 54all:: audio/tracker/tracker-5.3 55\&.PHONY: audio/tracker/tracker-5.3 56audio/tracker/tracker-5.3: tracker-5.3.tgz 57 58tracker-5.3.tgz: $F 59 @MAINTAINER="espie@openbsd.org" \\ 60 SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ " \\ 61 CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323" \\ 62 exec ${FETCH} "$@" 63 64 65.Ed 66This Makefile is usually invoked by the user from the directory where 67they wish to mirror distfiles, with variables 68.Ev FETCH 69and 70.Ev F 71set on the command line, e.g., 72.Bd -literal -offset indent 73cd mirror && make -j 5 -f path_to_makefile FETCH=fetch_script 74.Ed 75.Pp 76Targets are set up so that each port is referenced by its full name, and 77retrieves all its distfiles. 78The default 79.Ar all 80target is used to retrieve all distfiles. 81Targets 82.Ar ftp 83and 84.Ar cdrom 85can be used to retrieve all distfiles necessary to build ftp and cdrom 86packages, respectively. 87Actual fetching is usually invoked with a parallel-make option, so that 88several retrievals through ftp can proceed simultaneously. 89.Pp 90The 91.Ev F 92variable can be set to a dummy name, or a recent filename, to force 93re-fetching of anything which is older than the filename. 94Its intended use is to force re-fetching existing files, 95or to checksum all files. 96.Pp 97The 98.Ev ${FETCH} 99script should be supplied by the user, and will download and verify the 100archive file. 101It must obey the following variables: 102.Bl -tag -width DIST_SUBDIR 103.It Ev MAINTAINER 104Port maintainer, used to report errors, 105.It Ev ERROR 106Some ports problems can be detected while building the Makefile, in which 107case this variable will be set to a proper error message. 108.It Ev DIST_SUBDIR 109See 110.Xr ports 7 111for more details. 112The 113.Ev ${FETCH} 114script is responsible for creating this subdirectory and cd'ing to it 115before performing the actual fetch. 116.It Ev SITES 117A list of sites to try for fetching the distribution file. 118.It Ev CIPHER 119The checksumming utility to use for verifying the distribution file. 120It will normally be set to 121.Xr sha1 1 122unless you tinker with 123.Ev PREFERRED_CIPHER 124while building the mirroring Makefile. 125.It Ev CKSUM 126The corresponding checksum. 127If neither 128.Ev CIPHER 129nor 130.Ev CKSUM 131nor 132.Ev ERROR 133are set, the distribution file needs not be checked. 134.El 135.Pp 136A standard fetch strategy is to try all sites in order: whenever the 137distribution file is found, download it; verify the checksum; erase the 138file and try the next site if it doesn't match. 139.Pp 140Mirroring sites should update their master Makefile fairly often. 141Activities a proper mirror should offer (in order of decreasing importance): 142.Bl -tag -width XXXXXXXXX 143.It Mirror new files 144Use a proper fetch script to download missing files, 145.It Run Pa ${PORTSDIR}/infrastructure/fetch/link-checksums 146This script creates permanent hardlinks that preserve distfiles against 147checksum changes. 148.It Verify all checksums 149All checksums should be verified from time to time, and maintainers 150notified of persistent discrepancies, 151.It Check mastersites liveliness 152Use a tool such as 153.Sq mirror 154to check that the master sites haven't fallen 155off the Earth. 156Even though the first site in the site list is the 157most important site, good mirrors will scan all sites and report all 158problems, 159.It Remove old files 160To gain room this, the mirror should maintain a list of 161.Sq active 162files (easy enough, just provide a fetch script that just lists the 163file names), and remove files that are no longer active. 164Since 165.Ox 166releases happen every six months, this delay should be longer than that. 167.El 168Sample scripts are provided in the 169.Pa ${PORTSDIR}/infrastructure/fetch 170directory. 171.Sh FILES 172.Bl -tag -width XXXXXXXXX -compact 173.It Pa ${DISTDIR}/Makefile 174Main mirroring Makefile 175.It Pa ${PORTSDIR}/infrastructure/fetch/fetch-all 176Sample script usable to retrieve distfiles. 177.It Pa ${PORTSDIR}/infrastructure/fetch/check-all 178Sample script to check all distfiles checksums. 179.It Pa ${PORTSDIR}/infrastructure/fetch/link-checksums 180Populating checksums subdirectories with links, to guard against shifting 181checksums. 182.El 183.Sh SEE ALSO 184.Xr ports 7 185.Sh HISTORY 186This infrastructure was introduced for 187.Ox 2.7 188by Marc Espie, with feedback from Bob Beck, Todd Fries, Camiel Dobbelaar, 189and a few other people. 190.Sh CAVEATS 191Changing checksums is a recurring problem that is outside the direct 192control of the 193.Ox 194Project. 195Some software distributors change distribution files without 196warning, without changing the file name proper. 197Once the problem has been identified, the port maintainer should usually 198contact the software author to fix the problem, or, if the software author 199is unresponsive, the maintainer should use 200.Ev DIST_SUBDIR 201to provide some state to guard against shifting checksums. 202.Pp 203However, a more robust approach is also needed, so that ports users can 204depend on distfiles mirrors to carry what they need irregardless of those 205synchronization issues. 206The 207.Pa link-checksums 208script creates another access to the distfiles, indexed through the actual 209checksums that the files should match. 210Provided mirroring is run sufficiently often, together with 211.Pa link-checksums , 212two versions of the same distfile with respective checksums cksum1 and cksum2 213will be available under the names 214.Pa ${DISTFILES}/sha1/cksum1/distfile 215and 216.Pa ${DISTFILES}/sha1/cksum2/distfile . 217.Pp 218Starting revision 1.281, if 219.Ev REFETCH 220is set to true, 221.Pa bsd.port.mk 222will try to retrieve files under that naming scheme as a last resort. 223