1.\" Copyright (c) 2003-2009 Tim Kientzle 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd March 18, 2012 28.Dt LIBARCHIVE-FORMATS 5 29.Os 30.Sh NAME 31.Nm libarchive-formats 32.Nd archive formats supported by the libarchive library 33.Sh DESCRIPTION 34The 35.Xr libarchive 3 36library reads and writes a variety of streaming archive formats. 37Generally speaking, all of these archive formats consist of a series of 38.Dq entries . 39Each entry stores a single file system object, such as a file, directory, 40or symbolic link. 41.Pp 42The following provides a brief description of each format supported 43by libarchive, with some information about recognized extensions or 44limitations of the current library support. 45Note that just because a format is supported by libarchive does not 46imply that a program that uses libarchive will support that format. 47Applications that use libarchive specify which formats they wish 48to support, though many programs do use libarchive convenience 49functions to enable all supported formats. 50.Ss Tar Formats 51The 52.Xr libarchive 3 53library can read most tar archives. 54It can write POSIX-standard 55.Dq ustar 56and 57.Dq pax interchange 58formats as well as v7 tar format and a subset of the legacy GNU tar format. 59.Pp 60All tar formats store each entry in one or more 512-byte records. 61The first record is used for file metadata, including filename, 62timestamp, and mode information, and the file data is stored in 63subsequent records. 64Later variants have extended this by either appropriating undefined 65areas of the header record, extending the header to multiple records, 66or by storing special entries that modify the interpretation of 67subsequent entries. 68.Pp 69.Bl -tag -width indent 70.It Cm gnutar 71The 72.Xr libarchive 3 73library can read most GNU-format tar archives. 74It currently supports the most popular GNU extensions, including 75modern long filename and linkname support, as well as atime and ctime data. 76The libarchive library does not support multi-volume 77archives, nor the old GNU long filename format. 78It can read GNU sparse file entries, including the new POSIX-based 79formats. 80.Pp 81The 82.Xr libarchive 3 83library can write GNU tar format, including long filename 84and linkname support, as well as atime and ctime data. 85.It Cm pax 86The 87.Xr libarchive 3 88library can read and write POSIX-compliant pax interchange format 89archives. 90Pax interchange format archives are an extension of the older ustar 91format that adds a separate entry with additional attributes stored 92as key/value pairs immediately before each regular entry. 93The presence of these additional entries is the only difference between 94pax interchange format and the older ustar format. 95The extended attributes are of unlimited length and are stored 96as UTF-8 Unicode strings. 97Keywords defined in the standard are in all lowercase; vendors are allowed 98to define custom keys by preceding them with the vendor name in all uppercase. 99When writing pax archives, libarchive uses many of the SCHILY keys 100defined by Joerg Schilling's 101.Dq star 102archiver and a few LIBARCHIVE keys. 103The libarchive library can read most of the SCHILY keys 104and most of the GNU keys introduced by GNU tar. 105It silently ignores any keywords that it does not understand. 106.Pp 107The pax interchange format converts filenames to Unicode 108and stores them using the UTF-8 encoding. 109Prior to libarchive 3.0, libarchive erroneously assumed 110that the system wide-character routines natively supported 111Unicode. 112This caused it to mis-handle non-ASCII filenames on systems 113that did not satisfy this assumption. 114.It Cm restricted pax 115The libarchive library can also write pax archives in which it 116attempts to suppress the extended attributes entry whenever 117possible. 118The result will be identical to a ustar archive unless the 119extended attributes entry is required to store a long file 120name, long linkname, extended ACL, file flags, or if any of the standard 121ustar data (user name, group name, UID, GID, etc) cannot be fully 122represented in the ustar header. 123In all cases, the result can be dearchived by any program that 124can read POSIX-compliant pax interchange format archives. 125Programs that correctly read ustar format (see below) will also be 126able to read this format; any extended attributes will be extracted as 127separate files stored in 128.Pa PaxHeader 129directories. 130.It Cm ustar 131The libarchive library can both read and write this format. 132This format has the following limitations: 133.Bl -bullet -compact 134.It 135Device major and minor numbers are limited to 21 bits. 136Nodes with larger numbers will not be added to the archive. 137.It 138Path names in the archive are limited to 255 bytes. 139(Shorter if there is no / character in exactly the right place.) 140.It 141Symbolic links and hard links are stored in the archive with 142the name of the referenced file. 143This name is limited to 100 bytes. 144.It 145Extended attributes, file flags, and other extended 146security information cannot be stored. 147.It 148Archive entries are limited to 8 gigabytes in size. 149.El 150Note that the pax interchange format has none of these restrictions. 151The ustar format is old and widely supported. 152It is recommended when compatibility is the primary concern. 153.It Cm v7 154The libarchive library can read and write the legacy v7 tar format. 155This format has the following limitations: 156.Bl -bullet -compact 157.It 158Only regular files, directories, and symbolic links can be archived. 159Block and character device nodes, FIFOs, and sockets cannot be archived. 160.It 161Path names in the archive are limited to 100 bytes. 162.It 163Symbolic links and hard links are stored in the archive with 164the name of the referenced file. 165This name is limited to 100 bytes. 166.It 167User and group information are stored as numeric IDs; there 168is no provision for storing user or group names. 169.It 170Extended attributes, file flags, and other extended 171security information cannot be stored. 172.It 173Archive entries are limited to 8 gigabytes in size. 174.El 175Generally, users should prefer the ustar format for portability 176as the v7 tar format is both less useful and less portable. 177.El 178.Pp 179The libarchive library also reads a variety of commonly-used extensions to 180the basic tar format. 181These extensions are recognized automatically whenever they appear. 182.Bl -tag -width indent 183.It Numeric extensions. 184The POSIX standards require fixed-length numeric fields to be written with 185some character position reserved for terminators. 186Libarchive allows these fields to be written without terminator characters. 187This extends the allowable range; in particular, ustar archives with this 188extension can support entries up to 64 gigabytes in size. 189Libarchive also recognizes base-256 values in most numeric fields. 190This essentially removes all limitations on file size, modification time, 191and device numbers. 192.It Solaris extensions 193Libarchive recognizes ACL and extended attribute records written 194by Solaris tar. 195Currently, libarchive only has support for old-style ACLs; the 196newer NFSv4 ACLs are recognized but discarded. 197.El 198.Pp 199The first tar program appeared in Seventh Edition Unix in 1979. 200The first official standard for the tar file format was the 201.Dq ustar 202(Unix Standard Tar) format defined by POSIX in 1988. 203POSIX.1-2001 extended the ustar format to create the 204.Dq pax interchange 205format. 206.Ss Cpio Formats 207The libarchive library can read a number of common cpio variants and can write 208.Dq odc 209and 210.Dq newc 211format archives. 212A cpio archive stores each entry as a fixed-size header followed 213by a variable-length filename and variable-length data. 214Unlike the tar format, the cpio format does only minimal padding 215of the header or file data. 216There are several cpio variants, which differ primarily in 217how they store the initial header: some store the values as 218octal or hexadecimal numbers in ASCII, others as binary values of 219varying byte order and length. 220.Bl -tag -width indent 221.It Cm binary 222The libarchive library transparently reads both big-endian and little-endian 223variants of the original binary cpio format. 224This format used 32-bit binary values for file size and mtime, 225and 16-bit binary values for the other fields. 226.It Cm odc 227The libarchive library can both read and write this 228POSIX-standard format, which is officially known as the 229.Dq cpio interchange format 230or the 231.Dq octet-oriented cpio archive format 232and sometimes unofficially referred to as the 233.Dq old character format . 234This format stores the header contents as octal values in ASCII. 235It is standard, portable, and immune from byte-order confusion. 236File sizes and mtime are limited to 33 bits (8GB file size), 237other fields are limited to 18 bits. 238.It Cm SVR4/newc 239The libarchive library can read both CRC and non-CRC variants of 240this format. 241The SVR4 format uses eight-digit hexadecimal values for 242all header fields. 243This limits file size to 4GB, and also limits the mtime and 244other fields to 32 bits. 245The SVR4 format can optionally include a CRC of the file 246contents, although libarchive does not currently verify this CRC. 247.El 248.Pp 249Cpio first appeared in PWB/UNIX 1.0, which was released within 250AT&T in 1977. 251PWB/UNIX 1.0 formed the basis of System III Unix, released outside 252of AT&T in 1981. 253This makes cpio older than tar, although cpio was not included 254in Version 7 AT&T Unix. 255As a result, the tar command became much better known in universities 256and research groups that used Version 7. 257The combination of the 258.Nm find 259and 260.Nm cpio 261utilities provided very precise control over file selection. 262Unfortunately, the format has many limitations that make it unsuitable 263for widespread use. 264Only the POSIX format permits files over 4GB, and its 18-bit 265limit for most other fields makes it unsuitable for modern systems. 266In addition, cpio formats only store numeric UID/GID values (not 267usernames and group names), which can make it very difficult to correctly 268transfer archives across systems with dissimilar user numbering. 269.Ss Shar Formats 270A 271.Dq shell archive 272is a shell script that, when executed on a POSIX-compliant 273system, will recreate a collection of file system objects. 274The libarchive library can write two different kinds of shar archives: 275.Bl -tag -width indent 276.It Cm shar 277The traditional shar format uses a limited set of POSIX 278commands, including 279.Xr echo 1 , 280.Xr mkdir 1 , 281and 282.Xr sed 1 . 283It is suitable for portably archiving small collections of plain text files. 284However, it is not generally well-suited for large archives 285(many implementations of 286.Xr sh 1 287have limits on the size of a script) nor should it be used with non-text files. 288.It Cm shardump 289This format is similar to shar but encodes files using 290.Xr uuencode 1 291so that the result will be a plain text file regardless of the file contents. 292It also includes additional shell commands that attempt to reproduce as 293many file attributes as possible, including owner, mode, and flags. 294The additional commands used to restore file attributes make 295shardump archives less portable than plain shar archives. 296.El 297.Ss ISO9660 format 298Libarchive can read and extract from files containing ISO9660-compliant 299CDROM images. 300In many cases, this can remove the need to burn a physical CDROM 301just in order to read the files contained in an ISO9660 image. 302It also avoids security and complexity issues that come with 303virtual mounts and loopback devices. 304Libarchive supports the most common Rockridge extensions and has partial 305support for Joliet extensions. 306If both extensions are present, the Joliet extensions will be 307used and the Rockridge extensions will be ignored. 308In particular, this can create problems with hardlinks and symlinks, 309which are supported by Rockridge but not by Joliet. 310.Pp 311Libarchive reads ISO9660 images using a streaming strategy. 312This allows it to read compressed images directly 313(decompressing on the fly) and allows it to read images 314directly from network sockets, pipes, and other non-seekable 315data sources. 316This strategy works well for optimized ISO9660 images created 317by many popular programs. 318Such programs collect all directory information at the beginning 319of the ISO9660 image so it can be read from a physical disk 320with a minimum of seeking. 321However, not all ISO9660 images can be read in this fashion. 322.Pp 323Libarchive can also write ISO9660 images. 324Such images are fully optimized with the directory information 325preceding all file data. 326This is done by storing all file data to a temporary file 327while collecting directory information in memory. 328When the image is finished, libarchive writes out the 329directory structure followed by the file data. 330The location used for the temporary file can be changed 331by the usual environment variables. 332.Ss Zip format 333Libarchive can read and write zip format archives that have 334uncompressed entries and entries compressed with the 335.Dq deflate 336algorithm. 337Other zip compression algorithms are not supported. 338It can extract jar archives, archives that use Zip64 extensions and 339self-extracting zip archives. 340Libarchive can use either of two different strategies for 341reading Zip archives: 342a streaming strategy which is fast and can handle extremely 343large archives, and a seeking strategy which can correctly 344process self-extracting Zip archives and archives with 345deleted members or other in-place modifications. 346.Pp 347The streaming reader processes Zip archives as they are read. 348It can read archives of arbitrary size from tape or 349network sockets, and can decode Zip archives that have 350been separately compressed or encoded. 351However, self-extracting Zip archives and archives with 352certain types of modifications cannot be correctly 353handled. 354Such archives require that the reader first process the 355Central Directory, which is ordinarily located 356at the end of a Zip archive and is thus inaccessible 357to the streaming reader. 358If the program using libarchive has enabled seek support, then 359libarchive will use this to processes the central directory first. 360.Pp 361In particular, the seeking reader must be used to 362correctly handle self-extracting archives. 363Such archives consist of a program followed by a regular 364Zip archive. 365The streaming reader cannot parse the initial program 366portion, but the seeking reader starts by reading the 367Central Directory from the end of the archive. 368Similarly, Zip archives that have been modified in-place 369can have deleted entries or other garbage data that 370can only be accurately detected by first reading the 371Central Directory. 372.Ss Archive (library) file format 373The Unix archive format (commonly created by the 374.Xr ar 1 375archiver) is a general-purpose format which is 376used almost exclusively for object files to be 377read by the link editor 378.Xr ld 1 . 379The ar format has never been standardised. 380There are two common variants: 381the GNU format derived from SVR4, 382and the BSD format, which first appeared in 4.4BSD. 383The two differ primarily in their handling of filenames 384longer than 15 characters: 385the GNU/SVR4 variant writes a filename table at the beginning of the archive; 386the BSD format stores each long filename in an extension 387area adjacent to the entry. 388Libarchive can read both extensions, 389including archives that may include both types of long filenames. 390Programs using libarchive can write GNU/SVR4 format 391if they provide an entry called 392.Pa // 393containing a filename table to be written into the archive 394before any of the entries. 395Any entries whose names are not in the filename table 396will be written using BSD-style long filenames. 397This can cause problems for programs such as 398GNU ld that do not support the BSD-style long filenames. 399.Ss mtree 400Libarchive can read and write files in 401.Xr mtree 5 402format. 403This format is not a true archive format, but rather a textual description 404of a file hierarchy in which each line specifies the name of a file and 405provides specific metadata about that file. 406Libarchive can read all of the keywords supported by both 407the NetBSD and FreeBSD versions of 408.Xr mtree 8 , 409although many of the keywords cannot currently be stored in an 410.Tn archive_entry 411object. 412When writing, libarchive supports use of the 413.Xr archive_write_set_options 3 414interface to specify which keywords should be included in the 415output. 416If libarchive was compiled with access to suitable 417cryptographic libraries (such as the OpenSSL libraries), 418it can compute hash entries such as 419.Cm sha512 420or 421.Cm md5 422from file data being written to the mtree writer. 423.Pp 424When reading an mtree file, libarchive will locate the corresponding 425files on disk using the 426.Cm contents 427keyword if present or the regular filename. 428If it can locate and open the file on disk, it will use that 429to fill in any metadata that is missing from the mtree file 430and will read the file contents and return those to the program 431using libarchive. 432If it cannot locate and open the file on disk, libarchive 433will return an error for any attempt to read the entry 434body. 435.Ss 7-Zip 436Libarchive can read and write 7-Zip format archives. 437TODO: Need more information 438.Ss CAB 439Libarchive can read Microsoft Cabinet ( 440.Dq CAB ) 441format archives. 442TODO: Need more information. 443.Ss LHA 444TODO: Information about libarchive's LHA support 445.Ss RAR 446Libarchive has limited support for reading RAR format archives. 447Currently, libarchive can read RARv3 format archives 448which have been either created uncompressed, or compressed using 449any of the compression methods supported by the RARv3 format. 450Libarchive can also read self-extracting RAR archives. 451.Ss Warc 452Libarchive can read and write 453.Dq web archives . 454TODO: Need more information 455.Ss XAR 456Libarchive can read and write the XAR format used by many Apple tools. 457TODO: Need more information 458.Sh SEE ALSO 459.Xr ar 1 , 460.Xr cpio 1 , 461.Xr mkisofs 1 , 462.Xr shar 1 , 463.Xr tar 1 , 464.Xr zip 1 , 465.Xr zlib 3 , 466.Xr cpio 5 , 467.Xr mtree 5 , 468.Xr tar 5 469