1.\" Copyright (c) 2003-2009 Tim Kientzle 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd December 27, 2016 28.Dt LIBARCHIVE-FORMATS 5 29.Os 30.Sh NAME 31.Nm libarchive-formats 32.Nd archive formats supported by the libarchive library 33.Sh DESCRIPTION 34The 35.Xr libarchive 3 36library reads and writes a variety of streaming archive formats. 37Generally speaking, all of these archive formats consist of a series of 38.Dq entries . 39Each entry stores a single file system object, such as a file, directory, 40or symbolic link. 41.Pp 42The following provides a brief description of each format supported 43by libarchive, with some information about recognized extensions or 44limitations of the current library support. 45Note that just because a format is supported by libarchive does not 46imply that a program that uses libarchive will support that format. 47Applications that use libarchive specify which formats they wish 48to support, though many programs do use libarchive convenience 49functions to enable all supported formats. 50.Ss Tar Formats 51The 52.Xr libarchive 3 53library can read most tar archives. 54It can write POSIX-standard 55.Dq ustar 56and 57.Dq pax interchange 58formats as well as v7 tar format and a subset of the legacy GNU tar format. 59.Pp 60All tar formats store each entry in one or more 512-byte records. 61The first record is used for file metadata, including filename, 62timestamp, and mode information, and the file data is stored in 63subsequent records. 64Later variants have extended this by either appropriating undefined 65areas of the header record, extending the header to multiple records, 66or by storing special entries that modify the interpretation of 67subsequent entries. 68.Bl -tag -width indent 69.It Cm gnutar 70The 71.Xr libarchive 3 72library can read most GNU-format tar archives. 73It currently supports the most popular GNU extensions, including 74modern long filename and linkname support, as well as atime and ctime data. 75The libarchive library does not support multi-volume 76archives, nor the old GNU long filename format. 77It can read GNU sparse file entries, including the new POSIX-based 78formats. 79.Pp 80The 81.Xr libarchive 3 82library can write GNU tar format, including long filename 83and linkname support, as well as atime and ctime data. 84.It Cm pax 85The 86.Xr libarchive 3 87library can read and write POSIX-compliant pax interchange format 88archives. 89Pax interchange format archives are an extension of the older ustar 90format that adds a separate entry with additional attributes stored 91as key/value pairs immediately before each regular entry. 92The presence of these additional entries is the only difference between 93pax interchange format and the older ustar format. 94The extended attributes are of unlimited length and are stored 95as UTF-8 Unicode strings. 96Keywords defined in the standard are in all lowercase; vendors are allowed 97to define custom keys by preceding them with the vendor name in all uppercase. 98When writing pax archives, libarchive uses many of the SCHILY keys 99defined by Joerg Schilling's 100.Dq star 101archiver and a few LIBARCHIVE keys. 102The libarchive library can read most of the SCHILY keys 103and most of the GNU keys introduced by GNU tar. 104It silently ignores any keywords that it does not understand. 105.Pp 106The pax interchange format converts filenames to Unicode 107and stores them using the UTF-8 encoding. 108Prior to libarchive 3.0, libarchive erroneously assumed 109that the system wide-character routines natively supported 110Unicode. 111This caused it to mis-handle non-ASCII filenames on systems 112that did not satisfy this assumption. 113.It Cm restricted pax 114The libarchive library can also write pax archives in which it 115attempts to suppress the extended attributes entry whenever 116possible. 117The result will be identical to a ustar archive unless the 118extended attributes entry is required to store a long file 119name, long linkname, extended ACL, file flags, or if any of the standard 120ustar data (user name, group name, UID, GID, etc) cannot be fully 121represented in the ustar header. 122In all cases, the result can be dearchived by any program that 123can read POSIX-compliant pax interchange format archives. 124Programs that correctly read ustar format (see below) will also be 125able to read this format; any extended attributes will be extracted as 126separate files stored in 127.Pa PaxHeader 128directories. 129.It Cm ustar 130The libarchive library can both read and write this format. 131This format has the following limitations: 132.Bl -bullet -compact 133.It 134Device major and minor numbers are limited to 21 bits. 135Nodes with larger numbers will not be added to the archive. 136.It 137Path names in the archive are limited to 255 bytes. 138(Shorter if there is no / character in exactly the right place.) 139.It 140Symbolic links and hard links are stored in the archive with 141the name of the referenced file. 142This name is limited to 100 bytes. 143.It 144Extended attributes, file flags, and other extended 145security information cannot be stored. 146.It 147Archive entries are limited to 8 gigabytes in size. 148.El 149Note that the pax interchange format has none of these restrictions. 150The ustar format is old and widely supported. 151It is recommended when compatibility is the primary concern. 152.It Cm v7 153The libarchive library can read and write the legacy v7 tar format. 154This format has the following limitations: 155.Bl -bullet -compact 156.It 157Only regular files, directories, and symbolic links can be archived. 158Block and character device nodes, FIFOs, and sockets cannot be archived. 159.It 160Path names in the archive are limited to 100 bytes. 161.It 162Symbolic links and hard links are stored in the archive with 163the name of the referenced file. 164This name is limited to 100 bytes. 165.It 166User and group information are stored as numeric IDs; there 167is no provision for storing user or group names. 168.It 169Extended attributes, file flags, and other extended 170security information cannot be stored. 171.It 172Archive entries are limited to 8 gigabytes in size. 173.El 174Generally, users should prefer the ustar format for portability 175as the v7 tar format is both less useful and less portable. 176.El 177.Pp 178The libarchive library also reads a variety of commonly-used extensions to 179the basic tar format. 180These extensions are recognized automatically whenever they appear. 181.Bl -tag -width indent 182.It Numeric extensions. 183The POSIX standards require fixed-length numeric fields to be written with 184some character position reserved for terminators. 185Libarchive allows these fields to be written without terminator characters. 186This extends the allowable range; in particular, ustar archives with this 187extension can support entries up to 64 gigabytes in size. 188Libarchive also recognizes base-256 values in most numeric fields. 189This essentially removes all limitations on file size, modification time, 190and device numbers. 191.It Solaris extensions 192Libarchive recognizes ACL and extended attribute records written 193by Solaris tar. 194.El 195.Pp 196The first tar program appeared in Seventh Edition Unix in 1979. 197The first official standard for the tar file format was the 198.Dq ustar 199(Unix Standard Tar) format defined by POSIX in 1988. 200POSIX.1-2001 extended the ustar format to create the 201.Dq pax interchange 202format. 203.Ss Cpio Formats 204The libarchive library can read and write a number of common cpio 205variants. A cpio archive stores each entry as a fixed-size header 206followed by a variable-length filename and variable-length data. 207Unlike the tar format, the cpio format does only minimal padding of 208the header or file data. There are several cpio variants, which 209differ primarily in how they store the initial header: some store the 210values as octal or hexadecimal numbers in ASCII, others as binary 211values of varying byte order and length. 212.Bl -tag -width indent 213.It Cm binary 214The libarchive library transparently reads both big-endian and 215little-endian variants of the the two binary cpio formats; the 216original one from PWB/UNIX, and the later, more widely used, variant. 217This format used 32-bit binary values for file size and mtime, and 21816-bit binary values for the other fields. The formats support only 219the file types present in UNIX at the time of their creation. File 220sizes are limited to 24 bits in the PWB format, because of the limits 221of the file system, and to 31 bits in the newer binary format, where 222signed 32 bit longs were used. 223.It Cm odc 224This is the POSIX standardized format, which is officially known as the 225.Dq cpio interchange format 226or the 227.Dq octet-oriented cpio archive format 228and sometimes unofficially referred to as the 229.Dq old character format . 230This format stores the header contents as octal values in ASCII. 231It is standard, portable, and immune from byte-order confusion. 232File sizes and mtime are limited to 33 bits (8GB file size), 233other fields are limited to 18 bits. 234.It Cm SVR4/newc 235The libarchive library can read both CRC and non-CRC variants of 236this format. 237The SVR4 format uses eight-digit hexadecimal values for 238all header fields. 239This limits file size to 4GB, and also limits the mtime and 240other fields to 32 bits. 241The SVR4 format can optionally include a CRC of the file 242contents, although libarchive does not currently verify this CRC. 243.El 244.Pp 245Cpio first appeared in PWB/UNIX 1.0, which was released within 246AT&T in 1977. 247PWB/UNIX 1.0 formed the basis of System III Unix, released outside 248of AT&T in 1981. 249This makes cpio older than tar, although cpio was not included 250in Version 7 AT&T Unix. 251As a result, the tar command became much better known in universities 252and research groups that used Version 7. 253The combination of the 254.Nm find 255and 256.Nm cpio 257utilities provided very precise control over file selection. 258Unfortunately, the format has many limitations that make it unsuitable 259for widespread use. 260Only the POSIX format permits files over 4GB, and its 18-bit 261limit for most other fields makes it unsuitable for modern systems. 262In addition, cpio formats only store numeric UID/GID values (not 263usernames and group names), which can make it very difficult to correctly 264transfer archives across systems with dissimilar user numbering. 265.Ss Shar Formats 266A 267.Dq shell archive 268is a shell script that, when executed on a POSIX-compliant 269system, will recreate a collection of file system objects. 270The libarchive library can write two different kinds of shar archives: 271.Bl -tag -width indent 272.It Cm shar 273The traditional shar format uses a limited set of POSIX 274commands, including 275.Xr echo 1 , 276.Xr mkdir 1 , 277and 278.Xr sed 1 . 279It is suitable for portably archiving small collections of plain text files. 280However, it is not generally well-suited for large archives 281(many implementations of 282.Xr sh 1 283have limits on the size of a script) nor should it be used with non-text files. 284.It Cm shardump 285This format is similar to shar but encodes files using 286.Xr uuencode 1 287so that the result will be a plain text file regardless of the file contents. 288It also includes additional shell commands that attempt to reproduce as 289many file attributes as possible, including owner, mode, and flags. 290The additional commands used to restore file attributes make 291shardump archives less portable than plain shar archives. 292.El 293.Ss ISO9660 format 294Libarchive can read and extract from files containing ISO9660-compliant 295CDROM images. 296In many cases, this can remove the need to burn a physical CDROM 297just in order to read the files contained in an ISO9660 image. 298It also avoids security and complexity issues that come with 299virtual mounts and loopback devices. 300Libarchive supports the most common Rockridge extensions and has partial 301support for Joliet extensions. 302If both extensions are present, the Joliet extensions will be 303used and the Rockridge extensions will be ignored. 304In particular, this can create problems with hardlinks and symlinks, 305which are supported by Rockridge but not by Joliet. 306.Pp 307Libarchive reads ISO9660 images using a streaming strategy. 308This allows it to read compressed images directly 309(decompressing on the fly) and allows it to read images 310directly from network sockets, pipes, and other non-seekable 311data sources. 312This strategy works well for optimized ISO9660 images created 313by many popular programs. 314Such programs collect all directory information at the beginning 315of the ISO9660 image so it can be read from a physical disk 316with a minimum of seeking. 317However, not all ISO9660 images can be read in this fashion. 318.Pp 319Libarchive can also write ISO9660 images. 320Such images are fully optimized with the directory information 321preceding all file data. 322This is done by storing all file data to a temporary file 323while collecting directory information in memory. 324When the image is finished, libarchive writes out the 325directory structure followed by the file data. 326The location used for the temporary file can be changed 327by the usual environment variables. 328.Ss Zip format 329Libarchive can read and write zip format archives that have 330uncompressed entries and entries compressed with the 331.Dq deflate 332algorithm. 333Other zip compression algorithms are not supported. 334It can extract jar archives, archives that use Zip64 extensions and 335self-extracting zip archives. 336Libarchive can use either of two different strategies for 337reading Zip archives: 338a streaming strategy which is fast and can handle extremely 339large archives, and a seeking strategy which can correctly 340process self-extracting Zip archives and archives with 341deleted members or other in-place modifications. 342.Pp 343The streaming reader processes Zip archives as they are read. 344It can read archives of arbitrary size from tape or 345network sockets, and can decode Zip archives that have 346been separately compressed or encoded. 347However, self-extracting Zip archives and archives with 348certain types of modifications cannot be correctly 349handled. 350Such archives require that the reader first process the 351Central Directory, which is ordinarily located 352at the end of a Zip archive and is thus inaccessible 353to the streaming reader. 354If the program using libarchive has enabled seek support, then 355libarchive will use this to processes the central directory first. 356.Pp 357In particular, the seeking reader must be used to 358correctly handle self-extracting archives. 359Such archives consist of a program followed by a regular 360Zip archive. 361The streaming reader cannot parse the initial program 362portion, but the seeking reader starts by reading the 363Central Directory from the end of the archive. 364Similarly, Zip archives that have been modified in-place 365can have deleted entries or other garbage data that 366can only be accurately detected by first reading the 367Central Directory. 368.Ss Archive (library) file format 369The Unix archive format (commonly created by the 370.Xr ar 1 371archiver) is a general-purpose format which is 372used almost exclusively for object files to be 373read by the link editor 374.Xr ld 1 . 375The ar format has never been standardised. 376There are two common variants: 377the GNU format derived from SVR4, 378and the BSD format, which first appeared in 4.4BSD. 379The two differ primarily in their handling of filenames 380longer than 15 characters: 381the GNU/SVR4 variant writes a filename table at the beginning of the archive; 382the BSD format stores each long filename in an extension 383area adjacent to the entry. 384Libarchive can read both extensions, 385including archives that may include both types of long filenames. 386Programs using libarchive can write GNU/SVR4 format 387if they provide an entry called 388.Pa // 389containing a filename table to be written into the archive 390before any of the entries. 391Any entries whose names are not in the filename table 392will be written using BSD-style long filenames. 393This can cause problems for programs such as 394GNU ld that do not support the BSD-style long filenames. 395.Ss mtree 396Libarchive can read and write files in 397.Xr mtree 5 398format. 399This format is not a true archive format, but rather a textual description 400of a file hierarchy in which each line specifies the name of a file and 401provides specific metadata about that file. 402Libarchive can read all of the keywords supported by both 403the NetBSD and FreeBSD versions of 404.Xr mtree 8 , 405although many of the keywords cannot currently be stored in an 406.Tn archive_entry 407object. 408When writing, libarchive supports use of the 409.Xr archive_write_set_options 3 410interface to specify which keywords should be included in the 411output. 412If libarchive was compiled with access to suitable 413cryptographic libraries (such as the OpenSSL libraries), 414it can compute hash entries such as 415.Cm sha512 416or 417.Cm md5 418from file data being written to the mtree writer. 419.Pp 420When reading an mtree file, libarchive will locate the corresponding 421files on disk using the 422.Cm contents 423keyword if present or the regular filename. 424If it can locate and open the file on disk, it will use that 425to fill in any metadata that is missing from the mtree file 426and will read the file contents and return those to the program 427using libarchive. 428If it cannot locate and open the file on disk, libarchive 429will return an error for any attempt to read the entry 430body. 431.Ss 7-Zip 432Libarchive can read and write 7-Zip format archives. 433TODO: Need more information 434.Ss CAB 435Libarchive can read Microsoft Cabinet ( 436.Dq CAB ) 437format archives. 438TODO: Need more information. 439.Ss LHA 440TODO: Information about libarchive's LHA support 441.Ss RAR 442Libarchive has limited support for reading RAR format archives. 443Currently, libarchive can read RARv3 format archives 444which have been either created uncompressed, or compressed using 445any of the compression methods supported by the RARv3 format. 446Libarchive can also read self-extracting RAR archives. 447.Ss Warc 448Libarchive can read and write 449.Dq web archives . 450TODO: Need more information 451.Ss XAR 452Libarchive can read and write the XAR format used by many Apple tools. 453TODO: Need more information 454.Sh SEE ALSO 455.Xr ar 1 , 456.Xr cpio 1 , 457.Xr mkisofs 1 , 458.Xr shar 1 , 459.Xr tar 1 , 460.Xr zip 1 , 461.Xr zlib 3 , 462.Xr cpio 5 , 463.Xr mtree 5 , 464.Xr tar 5 465