1.\" Copyright (c) 2003-2009 Tim Kientzle 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd December 27, 2016 28.Dt LIBARCHIVE-FORMATS 5 29.Os 30.Sh NAME 31.Nm libarchive-formats 32.Nd archive formats supported by the libarchive library 33.Sh DESCRIPTION 34The 35.Xr libarchive 3 36library reads and writes a variety of streaming archive formats. 37Generally speaking, all of these archive formats consist of a series of 38.Dq entries . 39Each entry stores a single file system object, such as a file, directory, 40or symbolic link. 41.Pp 42The following provides a brief description of each format supported 43by libarchive, with some information about recognized extensions or 44limitations of the current library support. 45Note that just because a format is supported by libarchive does not 46imply that a program that uses libarchive will support that format. 47Applications that use libarchive specify which formats they wish 48to support, though many programs do use libarchive convenience 49functions to enable all supported formats. 50.Ss Tar Formats 51The 52.Xr libarchive 3 53library can read most tar archives. 54It can write POSIX-standard 55.Dq ustar 56and 57.Dq pax interchange 58formats as well as v7 tar format and a subset of the legacy GNU tar format. 59.Pp 60All tar formats store each entry in one or more 512-byte records. 61The first record is used for file metadata, including filename, 62timestamp, and mode information, and the file data is stored in 63subsequent records. 64Later variants have extended this by either appropriating undefined 65areas of the header record, extending the header to multiple records, 66or by storing special entries that modify the interpretation of 67subsequent entries. 68.Bl -tag -width indent 69.It Cm gnutar 70The 71.Xr libarchive 3 72library can read most GNU-format tar archives. 73It currently supports the most popular GNU extensions, including 74modern long filename and linkname support, as well as atime and ctime data. 75The libarchive library does not support multi-volume 76archives, nor the old GNU long filename format. 77It can read GNU sparse file entries, including the new POSIX-based 78formats. 79.Pp 80The 81.Xr libarchive 3 82library can write GNU tar format, including long filename 83and linkname support, as well as atime and ctime data. 84.It Cm pax 85The 86.Xr libarchive 3 87library can read and write POSIX-compliant pax interchange format 88archives. 89Pax interchange format archives are an extension of the older ustar 90format that adds a separate entry with additional attributes stored 91as key/value pairs immediately before each regular entry. 92The presence of these additional entries is the only difference between 93pax interchange format and the older ustar format. 94The extended attributes are of unlimited length and are stored 95as UTF-8 Unicode strings. 96Keywords defined in the standard are in all lowercase; vendors are allowed 97to define custom keys by preceding them with the vendor name in all uppercase. 98When writing pax archives, libarchive uses many of the SCHILY keys 99defined by Joerg Schilling's 100.Dq star 101archiver and a few LIBARCHIVE keys. 102The libarchive library can read most of the SCHILY keys 103and most of the GNU keys introduced by GNU tar. 104It silently ignores any keywords that it does not understand. 105.Pp 106The pax interchange format converts filenames to Unicode 107and stores them using the UTF-8 encoding. 108Prior to libarchive 3.0, libarchive erroneously assumed 109that the system wide-character routines natively supported 110Unicode. 111This caused it to mis-handle non-ASCII filenames on systems 112that did not satisfy this assumption. 113.It Cm restricted pax 114The libarchive library can also write pax archives in which it 115attempts to suppress the extended attributes entry whenever 116possible. 117The result will be identical to a ustar archive unless the 118extended attributes entry is required to store a long file 119name, long linkname, extended ACL, file flags, or if any of the standard 120ustar data (user name, group name, UID, GID, etc) cannot be fully 121represented in the ustar header. 122In all cases, the result can be dearchived by any program that 123can read POSIX-compliant pax interchange format archives. 124Programs that correctly read ustar format (see below) will also be 125able to read this format; any extended attributes will be extracted as 126separate files stored in 127.Pa PaxHeader 128directories. 129.It Cm ustar 130The libarchive library can both read and write this format. 131This format has the following limitations: 132.Bl -bullet -compact 133.It 134Device major and minor numbers are limited to 21 bits. 135Nodes with larger numbers will not be added to the archive. 136.It 137Path names in the archive are limited to 255 bytes. 138(Shorter if there is no / character in exactly the right place.) 139.It 140Symbolic links and hard links are stored in the archive with 141the name of the referenced file. 142This name is limited to 100 bytes. 143.It 144Extended attributes, file flags, and other extended 145security information cannot be stored. 146.It 147Archive entries are limited to 8 gigabytes in size. 148.El 149Note that the pax interchange format has none of these restrictions. 150The ustar format is old and widely supported. 151It is recommended when compatibility is the primary concern. 152.It Cm v7 153The libarchive library can read and write the legacy v7 tar format. 154This format has the following limitations: 155.Bl -bullet -compact 156.It 157Only regular files, directories, and symbolic links can be archived. 158Block and character device nodes, FIFOs, and sockets cannot be archived. 159.It 160Path names in the archive are limited to 100 bytes. 161.It 162Symbolic links and hard links are stored in the archive with 163the name of the referenced file. 164This name is limited to 100 bytes. 165.It 166User and group information are stored as numeric IDs; there 167is no provision for storing user or group names. 168.It 169Extended attributes, file flags, and other extended 170security information cannot be stored. 171.It 172Archive entries are limited to 8 gigabytes in size. 173.El 174Generally, users should prefer the ustar format for portability 175as the v7 tar format is both less useful and less portable. 176.El 177.Pp 178The libarchive library also reads a variety of commonly-used extensions to 179the basic tar format. 180These extensions are recognized automatically whenever they appear. 181.Bl -tag -width indent 182.It Numeric extensions. 183The POSIX standards require fixed-length numeric fields to be written with 184some character position reserved for terminators. 185Libarchive allows these fields to be written without terminator characters. 186This extends the allowable range; in particular, ustar archives with this 187extension can support entries up to 64 gigabytes in size. 188Libarchive also recognizes base-256 values in most numeric fields. 189This essentially removes all limitations on file size, modification time, 190and device numbers. 191.It Solaris extensions 192Libarchive recognizes ACL and extended attribute records written 193by Solaris tar. 194.El 195.Pp 196The first tar program appeared in Seventh Edition Unix in 1979. 197The first official standard for the tar file format was the 198.Dq ustar 199(Unix Standard Tar) format defined by POSIX in 1988. 200POSIX.1-2001 extended the ustar format to create the 201.Dq pax interchange 202format. 203.Ss Cpio Formats 204The libarchive library can read a number of common cpio variants and can write 205.Dq odc 206and 207.Dq newc 208format archives. 209A cpio archive stores each entry as a fixed-size header followed 210by a variable-length filename and variable-length data. 211Unlike the tar format, the cpio format does only minimal padding 212of the header or file data. 213There are several cpio variants, which differ primarily in 214how they store the initial header: some store the values as 215octal or hexadecimal numbers in ASCII, others as binary values of 216varying byte order and length. 217.Bl -tag -width indent 218.It Cm binary 219The libarchive library transparently reads both big-endian and little-endian 220variants of the original binary cpio format. 221This format used 32-bit binary values for file size and mtime, 222and 16-bit binary values for the other fields. 223.It Cm odc 224The libarchive library can both read and write this 225POSIX-standard format, which is officially known as the 226.Dq cpio interchange format 227or the 228.Dq octet-oriented cpio archive format 229and sometimes unofficially referred to as the 230.Dq old character format . 231This format stores the header contents as octal values in ASCII. 232It is standard, portable, and immune from byte-order confusion. 233File sizes and mtime are limited to 33 bits (8GB file size), 234other fields are limited to 18 bits. 235.It Cm SVR4/newc 236The libarchive library can read both CRC and non-CRC variants of 237this format. 238The SVR4 format uses eight-digit hexadecimal values for 239all header fields. 240This limits file size to 4GB, and also limits the mtime and 241other fields to 32 bits. 242The SVR4 format can optionally include a CRC of the file 243contents, although libarchive does not currently verify this CRC. 244.El 245.Pp 246Cpio first appeared in PWB/UNIX 1.0, which was released within 247AT&T in 1977. 248PWB/UNIX 1.0 formed the basis of System III Unix, released outside 249of AT&T in 1981. 250This makes cpio older than tar, although cpio was not included 251in Version 7 AT&T Unix. 252As a result, the tar command became much better known in universities 253and research groups that used Version 7. 254The combination of the 255.Nm find 256and 257.Nm cpio 258utilities provided very precise control over file selection. 259Unfortunately, the format has many limitations that make it unsuitable 260for widespread use. 261Only the POSIX format permits files over 4GB, and its 18-bit 262limit for most other fields makes it unsuitable for modern systems. 263In addition, cpio formats only store numeric UID/GID values (not 264usernames and group names), which can make it very difficult to correctly 265transfer archives across systems with dissimilar user numbering. 266.Ss Shar Formats 267A 268.Dq shell archive 269is a shell script that, when executed on a POSIX-compliant 270system, will recreate a collection of file system objects. 271The libarchive library can write two different kinds of shar archives: 272.Bl -tag -width indent 273.It Cm shar 274The traditional shar format uses a limited set of POSIX 275commands, including 276.Xr echo 1 , 277.Xr mkdir 1 , 278and 279.Xr sed 1 . 280It is suitable for portably archiving small collections of plain text files. 281However, it is not generally well-suited for large archives 282(many implementations of 283.Xr sh 1 284have limits on the size of a script) nor should it be used with non-text files. 285.It Cm shardump 286This format is similar to shar but encodes files using 287.Xr uuencode 1 288so that the result will be a plain text file regardless of the file contents. 289It also includes additional shell commands that attempt to reproduce as 290many file attributes as possible, including owner, mode, and flags. 291The additional commands used to restore file attributes make 292shardump archives less portable than plain shar archives. 293.El 294.Ss ISO9660 format 295Libarchive can read and extract from files containing ISO9660-compliant 296CDROM images. 297In many cases, this can remove the need to burn a physical CDROM 298just in order to read the files contained in an ISO9660 image. 299It also avoids security and complexity issues that come with 300virtual mounts and loopback devices. 301Libarchive supports the most common Rockridge extensions and has partial 302support for Joliet extensions. 303If both extensions are present, the Joliet extensions will be 304used and the Rockridge extensions will be ignored. 305In particular, this can create problems with hardlinks and symlinks, 306which are supported by Rockridge but not by Joliet. 307.Pp 308Libarchive reads ISO9660 images using a streaming strategy. 309This allows it to read compressed images directly 310(decompressing on the fly) and allows it to read images 311directly from network sockets, pipes, and other non-seekable 312data sources. 313This strategy works well for optimized ISO9660 images created 314by many popular programs. 315Such programs collect all directory information at the beginning 316of the ISO9660 image so it can be read from a physical disk 317with a minimum of seeking. 318However, not all ISO9660 images can be read in this fashion. 319.Pp 320Libarchive can also write ISO9660 images. 321Such images are fully optimized with the directory information 322preceding all file data. 323This is done by storing all file data to a temporary file 324while collecting directory information in memory. 325When the image is finished, libarchive writes out the 326directory structure followed by the file data. 327The location used for the temporary file can be changed 328by the usual environment variables. 329.Ss Zip format 330Libarchive can read and write zip format archives that have 331uncompressed entries and entries compressed with the 332.Dq deflate 333algorithm. 334Other zip compression algorithms are not supported. 335It can extract jar archives, archives that use Zip64 extensions and 336self-extracting zip archives. 337Libarchive can use either of two different strategies for 338reading Zip archives: 339a streaming strategy which is fast and can handle extremely 340large archives, and a seeking strategy which can correctly 341process self-extracting Zip archives and archives with 342deleted members or other in-place modifications. 343.Pp 344The streaming reader processes Zip archives as they are read. 345It can read archives of arbitrary size from tape or 346network sockets, and can decode Zip archives that have 347been separately compressed or encoded. 348However, self-extracting Zip archives and archives with 349certain types of modifications cannot be correctly 350handled. 351Such archives require that the reader first process the 352Central Directory, which is ordinarily located 353at the end of a Zip archive and is thus inaccessible 354to the streaming reader. 355If the program using libarchive has enabled seek support, then 356libarchive will use this to processes the central directory first. 357.Pp 358In particular, the seeking reader must be used to 359correctly handle self-extracting archives. 360Such archives consist of a program followed by a regular 361Zip archive. 362The streaming reader cannot parse the initial program 363portion, but the seeking reader starts by reading the 364Central Directory from the end of the archive. 365Similarly, Zip archives that have been modified in-place 366can have deleted entries or other garbage data that 367can only be accurately detected by first reading the 368Central Directory. 369.Ss Archive (library) file format 370The Unix archive format (commonly created by the 371.Xr ar 1 372archiver) is a general-purpose format which is 373used almost exclusively for object files to be 374read by the link editor 375.Xr ld 1 . 376The ar format has never been standardised. 377There are two common variants: 378the GNU format derived from SVR4, 379and the BSD format, which first appeared in 4.4BSD. 380The two differ primarily in their handling of filenames 381longer than 15 characters: 382the GNU/SVR4 variant writes a filename table at the beginning of the archive; 383the BSD format stores each long filename in an extension 384area adjacent to the entry. 385Libarchive can read both extensions, 386including archives that may include both types of long filenames. 387Programs using libarchive can write GNU/SVR4 format 388if they provide an entry called 389.Pa // 390containing a filename table to be written into the archive 391before any of the entries. 392Any entries whose names are not in the filename table 393will be written using BSD-style long filenames. 394This can cause problems for programs such as 395GNU ld that do not support the BSD-style long filenames. 396.Ss mtree 397Libarchive can read and write files in 398.Xr mtree 5 399format. 400This format is not a true archive format, but rather a textual description 401of a file hierarchy in which each line specifies the name of a file and 402provides specific metadata about that file. 403Libarchive can read all of the keywords supported by both 404the NetBSD and FreeBSD versions of 405.Xr mtree 8 , 406although many of the keywords cannot currently be stored in an 407.Tn archive_entry 408object. 409When writing, libarchive supports use of the 410.Xr archive_write_set_options 3 411interface to specify which keywords should be included in the 412output. 413If libarchive was compiled with access to suitable 414cryptographic libraries (such as the OpenSSL libraries), 415it can compute hash entries such as 416.Cm sha512 417or 418.Cm md5 419from file data being written to the mtree writer. 420.Pp 421When reading an mtree file, libarchive will locate the corresponding 422files on disk using the 423.Cm contents 424keyword if present or the regular filename. 425If it can locate and open the file on disk, it will use that 426to fill in any metadata that is missing from the mtree file 427and will read the file contents and return those to the program 428using libarchive. 429If it cannot locate and open the file on disk, libarchive 430will return an error for any attempt to read the entry 431body. 432.Ss 7-Zip 433Libarchive can read and write 7-Zip format archives. 434TODO: Need more information 435.Ss CAB 436Libarchive can read Microsoft Cabinet ( 437.Dq CAB ) 438format archives. 439TODO: Need more information. 440.Ss LHA 441TODO: Information about libarchive's LHA support 442.Ss RAR 443Libarchive has limited support for reading RAR format archives. 444Currently, libarchive can read RARv3 format archives 445which have been either created uncompressed, or compressed using 446any of the compression methods supported by the RARv3 format. 447Libarchive can also read self-extracting RAR archives. 448.Ss Warc 449Libarchive can read and write 450.Dq web archives . 451TODO: Need more information 452.Ss XAR 453Libarchive can read and write the XAR format used by many Apple tools. 454TODO: Need more information 455.Sh SEE ALSO 456.Xr ar 1 , 457.Xr cpio 1 , 458.Xr mkisofs 1 , 459.Xr shar 1 , 460.Xr tar 1 , 461.Xr zip 1 , 462.Xr zlib 3 , 463.Xr cpio 5 , 464.Xr mtree 5 , 465.Xr tar 5 466