1.\"- 2.\" Copyright (c) 2000-2014 Dag-Erling Smørgrav 3.\" Copyright (c) 2013-2016 Michael Gmelin <freebsd@grem.de> 4.\" All rights reserved. 5.\" Portions Copyright (c) 1999 Massachusetts Institute of Technology; used 6.\" by permission. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer 13.\" in this position and unchanged. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 3. The name of the author may not be used to endorse or promote products 18.\" derived from this software without specific prior written permission. 19.\" 20.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 21.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 22.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 23.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 25.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 29.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30.\" 31.Dd October 29, 2020 32.Dt FETCH 1 33.Os 34.Sh NAME 35.Nm fetch 36.Nd retrieve a file by Uniform Resource Locator 37.Sh SYNOPSIS 38.Nm 39.Op Fl 146AadFlMmnPpqRrsUv 40.Op Fl B Ar bytes 41.Op Fl -bind-address= Ns Ar host 42.Op Fl -ca-cert= Ns Ar file 43.Op Fl -ca-path= Ns Ar dir 44.Op Fl -cert= Ns Ar file 45.Op Fl -crl= Ns Ar file 46.Op Fl i Ar file 47.Op Fl -key= Ns Ar file 48.Op Fl N Ar file 49.Op Fl -no-passive 50.Op Fl -no-proxy= Ns Ar list 51.Op Fl -no-sslv3 52.Op Fl -no-tlsv1 53.Op Fl -no-verify-hostname 54.Op Fl -no-verify-peer 55.Op Fl o Ar file 56.Op Fl -referer= Ns Ar URL 57.Op Fl S Ar bytes 58.Op Fl T Ar seconds 59.Op Fl -user-agent= Ns Ar agent-string 60.Op Fl w Ar seconds 61.Ar URL ... 62.Nm 63.Op Fl 146AadFlMmnPpqRrsUv 64.Op Fl B Ar bytes 65.Op Fl -bind-address= Ns Ar host 66.Op Fl -ca-cert= Ns Ar file 67.Op Fl -ca-path= Ns Ar dir 68.Op Fl -cert= Ns Ar file 69.Op Fl -crl= Ns Ar file 70.Op Fl i Ar file 71.Op Fl -key= Ns Ar file 72.Op Fl N Ar file 73.Op Fl -no-passive 74.Op Fl -no-proxy= Ns Ar list 75.Op Fl -no-sslv3 76.Op Fl -no-tlsv1 77.Op Fl -no-verify-hostname 78.Op Fl -no-verify-peer 79.Op Fl o Ar file 80.Op Fl -referer= Ns Ar URL 81.Op Fl S Ar bytes 82.Op Fl T Ar seconds 83.Op Fl -user-agent= Ns Ar agent-string 84.Op Fl w Ar seconds 85.Fl h Ar host Fl f Ar file Oo Fl c Ar dir Oc 86.Sh DESCRIPTION 87The 88.Nm 89utility provides a command-line interface to the 90.Xr fetch 3 91library. 92Its purpose is to retrieve the file(s) pointed to by the URL(s) on the 93command line. 94.Pp 95The following options are available: 96.Bl -tag -width Fl 97.It Fl 1 , -one-file 98Stop and return exit code 0 at the first successfully retrieved file. 99.It Fl 4 , -ipv4-only 100Forces 101.Nm 102to use IPv4 addresses only. 103.It Fl 6 , -ipv6-only 104Forces 105.Nm 106to use IPv6 addresses only. 107.It Fl A , -no-redirect 108Do not automatically follow ``temporary'' (302) redirects. 109Some broken Web sites will return a redirect instead of a not-found 110error when the requested object does not exist. 111.It Fl a , -retry 112Automatically retry the transfer upon soft failures. 113.It Fl B Ar bytes , Fl -buffer-size= Ns Ar bytes 114Specify the read buffer size in bytes. 115The default is 16,384 bytes. 116Attempts to set a buffer size lower than this will be silently 117ignored. 118The number of reads actually performed is reported at verbosity level 119two or higher (see the 120.Fl v 121flag). 122.It Fl -bind-address= Ns Ar host 123Specifies a hostname or IP address to which sockets used for outgoing 124connections will be bound. 125.It Fl c Ar dir 126The file to retrieve is in directory 127.Ar dir 128on the remote host. 129This option is deprecated and is provided for backward compatibility 130only. 131.It Fl -ca-cert= Ns Ar file 132[SSL] 133Path to certificate bundle containing trusted CA certificates. 134If not specified, 135.Pa /usr/local/etc/ssl/cert.pem 136is used. 137If this file does not exist, 138.Pa /etc/ssl/cert.pem 139is used instead. 140If neither file exists and no CA path has been configured, 141OpenSSL's default CA cert and path settings apply. 142The certificate bundle can contain multiple CA certificates. 143The 144.Pa security/ca_root_nss 145port is a common source of a current CA bundle. 146.It Fl -ca-path= Ns Ar dir 147[SSL] 148The directory 149.Ar dir 150contains trusted CA hashes. 151.It Fl -cert= Ns Ar file 152[SSL] 153.Ar file 154is a PEM encoded client certificate/key which will be used in 155client certificate authentication. 156.It Fl -crl= Ns Ar file 157[SSL] 158Points to certificate revocation list 159.Ar file , 160which has to be in PEM format and may contain peer certificates that have 161been revoked. 162.It Fl d , -direct 163Use a direct connection even if a proxy is configured. 164.It Fl F , -force-restart 165In combination with the 166.Fl r 167flag, forces a restart even if the local and remote files have 168different modification times. 169Implies 170.Fl R . 171.It Fl f Ar file 172The file to retrieve is named 173.Ar file 174on the remote host. 175This option is deprecated and is provided for backward compatibility 176only. 177.It Fl h Ar host 178The file to retrieve is located on the host 179.Ar host . 180This option is deprecated and is provided for backward compatibility 181only. 182.It Fl i Ar file , Fl -if-modified-since= Ns Ar file 183If-Modified-Since mode: the remote file will only be retrieved if it 184is newer than 185.Ar file 186on the local host. 187(HTTP only) 188.It Fl -key= Ns Ar file 189[SSL] 190.Ar file 191is a PEM encoded client key that will be used in client certificate 192authentication in case key and client certificate are stored separately. 193.It Fl l , -symlink 194If the target is a file-scheme URL, make a symbolic link to the target 195rather than trying to copy it. 196.It Fl M 197.It Fl m , -mirror 198Mirror mode: if the file already exists locally and has the same size 199and modification time as the remote file, it will not be fetched. 200Note that the 201.Fl m 202and 203.Fl r 204flags are mutually exclusive. 205.It Fl N Ar file , Fl -netrc= Ns Ar file 206Use 207.Ar file 208instead of 209.Pa ~/.netrc 210to look up login names and passwords for FTP sites. 211See 212.Xr ftp 1 213for a description of the file format. 214This feature is experimental. 215.It Fl n , -no-mtime 216Do not preserve the modification time of the transferred file. 217.It Fl -no-passive 218Forces the FTP code to use active mode. 219.It Fl -no-proxy= Ns Ar list 220Either a single asterisk, which disables the use of proxies 221altogether, or a comma- or whitespace-separated list of hosts for 222which proxies should not be used. 223.It Fl -no-sslv3 224[SSL] 225Do not allow SSL version 3 when negotiating the connection. 226This option is deprecated and is provided for backward compatibility 227only. 228SSLv3 is disabled by default. 229Set 230.Ev SSL_ALLOW_SSL3 231to change this behavior. 232.It Fl -no-tlsv1 233[SSL] 234Do not allow TLS version 1 when negotiating the connection. 235.It Fl -no-verify-hostname 236[SSL] 237Do not verify that the hostname matches the subject of the 238certificate presented by the server. 239.It Fl -no-verify-peer 240[SSL] 241Do not verify the peer certificate against trusted CAs. 242.It Fl o Ar file , Fl -output= Ns Ar file 243Set the output file name to 244.Ar file . 245By default, a ``pathname'' is extracted from the specified URI, and 246its basename is used as the name of the output file. 247A 248.Ar file 249argument of 250.Sq Li \&- 251indicates that results are to be directed to the standard output. 252If the 253.Ar file 254argument is a directory, fetched file(s) will be placed within the 255directory, with name(s) selected as in the default behaviour. 256.It Fl P 257.It Fl p , -passive 258Use passive FTP. 259These flags have no effect, since passive FTP is the default, but are 260provided for compatibility with earlier versions where active FTP was 261the default. 262To force active mode, use the 263.Fl -no-passive 264flag or set the 265.Ev FTP_PASSIVE_MODE 266environment variable to 267.Ql NO . 268.It Fl -referer= Ns Ar URL 269Specifies the referrer URL to use for HTTP requests. 270If 271.Ar URL 272is set to 273.Dq auto , 274the document URL will be used as referrer URL. 275.It Fl q , -quiet 276Quiet mode. 277.It Fl R , -keep-output 278The output files are precious, and should not be deleted under any 279circumstances, even if the transfer failed or was incomplete. 280.It Fl r , -restart 281Restart a previously interrupted transfer. 282Note that the 283.Fl m 284and 285.Fl r 286flags are mutually exclusive. 287.It Fl S Ar bytes , Fl -require-size= Ns Ar bytes 288Require the file size reported by the server to match the specified 289value. 290If it does not, a message is printed and the file is not fetched. 291If the server does not support reporting file sizes, this option is 292ignored and the file is fetched unconditionally. 293.It Fl s , -print-size 294Print the size in bytes of each requested file, without fetching it. 295.It Fl T Ar seconds , Fl -timeout= Ns Ar seconds 296Set timeout value to 297.Ar seconds . 298Overrides the environment variables 299.Ev FTP_TIMEOUT 300for FTP transfers or 301.Ev HTTP_TIMEOUT 302for HTTP transfers if set. 303.It Fl U , -passive-portrange-default 304When using passive FTP, allocate the port for the data connection from 305the low (default) port range. 306See 307.Xr ip 4 308for details on how to specify which port range this corresponds to. 309.It Fl -user-agent= Ns Ar agent-string 310Specifies the User-Agent string to use for HTTP requests. 311This can be useful when working with HTTP origin or proxy servers that 312differentiate between user agents. 313.It Fl v , -verbose 314Increase verbosity level. 315.It Fl w Ar seconds , Fl -retry-delay= Ns Ar seconds 316When the 317.Fl a 318flag is specified, wait this many seconds between successive retries. 319.El 320.Pp 321If 322.Nm 323receives a 324.Dv SIGINFO 325signal (see the 326.Cm status 327argument for 328.Xr stty 1 ) , 329the current transfer rate statistics will be written to the 330standard error output, in the same format as the standard completion 331message. 332.Sh ENVIRONMENT 333.Bl -tag -width HTTP_TIMEOUT 334.It Ev FTP_TIMEOUT 335Maximum time, in seconds, to wait before aborting an FTP connection. 336.It Ev HTTP_TIMEOUT 337Maximum time, in seconds, to wait before aborting an HTTP connection. 338.El 339.Pp 340See 341.Xr fetch 3 342for a description of additional environment variables, including 343.Ev FETCH_BIND_ADDRESS , 344.Ev FTP_LOGIN , 345.Ev FTP_PASSIVE_MODE , 346.Ev FTP_PASSWORD , 347.Ev FTP_PROXY , 348.Ev ftp_proxy , 349.Ev HTTP_ACCEPT , 350.Ev HTTP_AUTH , 351.Ev HTTP_PROXY , 352.Ev http_proxy , 353.Ev HTTP_PROXY_AUTH , 354.Ev HTTP_REFERER , 355.Ev HTTP_USER_AGENT , 356.Ev NETRC , 357.Ev NO_PROXY , 358.Ev no_proxy , 359.Ev SSL_CA_CERT_FILE , 360.Ev SSL_CA_CERT_PATH , 361.Ev SSL_CLIENT_CERT_FILE , 362.Ev SSL_CLIENT_KEY_FILE , 363.Ev SSL_CRL_FILE , 364.Ev SSL_ALLOW_SSL3 , 365.Ev SSL_NO_TLS1 , 366.Ev SSL_NO_TLS1_1 , 367.Ev SSL_NO_TLS1_2 , 368.Ev SSL_NO_VERIFY_HOSTNAME 369and 370.Ev SSL_NO_VERIFY_PEER . 371.Sh EXIT STATUS 372The 373.Nm 374command returns zero on success, or one on failure. 375If multiple URLs are listed on the command line, 376.Nm 377will attempt to retrieve each one of them in turn, and will return 378zero only if they were all successfully retrieved. 379.Pp 380If the 381.Fl i 382argument is used and the remote file is not newer than the 383specified file then the command will still return success, 384although no file is transferred. 385.Sh EXAMPLES 386Silently try to fetch the URLs passed as parameters. 387The first one will fail. 388If the second URL succeeds the third one will not be tried: 389.Bd -literal -offset indent 390$ fetch -1 -q https://www.freebsd.org/bad.html \e 391 ftp.freebsd.org/pub/FreeBSD/README.TXT \e 392 https://www.fake.url 393fetch: https://www.freebsd.org/bad.html: Not Found 394.Ed 395.Pp 396Be verbose when retrieving the 397.Pa README.TXT 398file: 399.Bd -literal -offset indent 400$ fetch -v ftp.freebsd.org/pub/FreeBSD/README.TXT 401resolving server address: ftp.freebsd.org:80 402requesting http://ftp.freebsd.org/pub/FreeBSD/README.TXT 403local size / mtime: 4259 / 1431015519 404remote size / mtime: 4259 / 1431015519 405README.TXT 4259 B 44 MBps 00s 406.Ed 407.Pp 408Quietly save the 409.Pa README.TXT 410file as 411.Pa myreadme.txt 412and do not delete the output file under any circumstances: 413.Bd -literal -offset indent 414fetch -o myreadme.txt -q -R ftp.freebsd.org/pub/FreeBSD/README.TXT 415.Ed 416.Pp 417Print the size of the requested file and identify the request with a custom user 418agent string: 419.Bd -literal -offset indent 420$ fetch -s ftp.freebsd.org/pub/FreeBSD/README.TXT 421--user-agent="Mozilla/5.0 (X11; FreeBSD x86_64; rv:78.0) Gecko/20100101" 4223513231 423.Ed 424.Pp 425Restart the transfer of the 426.Pa README.TXT 427file and retry the transfer upon soft failures: 428.Bd -literal -offset indent 429$ fetch -a -r http://ftp.freebsd.org/pub/FreeBSD/README.TXT 430.Ed 431.Sh SEE ALSO 432.Xr fetch 3 , 433.Xr phttpget 8 434.Sh HISTORY 435The 436.Nm 437command appeared in 438.Fx 2.1.5 . 439This implementation first appeared in 440.Fx 4.1 . 441.Sh AUTHORS 442.An -nosplit 443The original implementation of 444.Nm 445was done by 446.An Jean-Marc Zucconi Aq Mt jmz@FreeBSD.org . 447It was extensively re-worked for 448.Fx 2.2 449by 450.An Garrett Wollman Aq Mt wollman@FreeBSD.org , 451and later completely rewritten to use the 452.Xr fetch 3 453library by 454.An Dag-Erling Sm\(/orgrav Aq Mt des@FreeBSD.org 455and 456.An Michael Gmelin Aq Mt freebsd@grem.de . 457.Sh NOTES 458The 459.Fl b 460and 461.Fl t 462options are no longer supported and will generate warnings. 463They were workarounds for bugs in other OSes which this implementation 464does not trigger. 465.Pp 466One cannot both use the 467.Fl h , 468.Fl c 469and 470.Fl f 471options and specify URLs on the command line. 472