1.\" $File: file.man,v 1.144 2021/02/05 22:08:31 christos Exp $ 2.Dd February 5, 2021 3.Dt FILE __CSECTION__ 4.Os 5.Sh NAME 6.Nm file 7.Nd determine file type 8.Sh SYNOPSIS 9.Nm 10.Bk -words 11.Op Fl bcdEhiklLNnprsSvzZ0 12.Op Fl Fl apple 13.Op Fl Fl exclude-quiet 14.Op Fl Fl extension 15.Op Fl Fl mime-encoding 16.Op Fl Fl mime-type 17.Op Fl e Ar testname 18.Op Fl F Ar separator 19.Op Fl f Ar namefile 20.Op Fl m Ar magicfiles 21.Op Fl P Ar name=value 22.Ar 23.Ek 24.Nm 25.Fl C 26.Op Fl m Ar magicfiles 27.Nm 28.Op Fl Fl help 29.Sh DESCRIPTION 30This manual page documents version __VERSION__ of the 31.Nm 32command. 33.Pp 34.Nm 35tests each argument in an attempt to classify it. 36There are three sets of tests, performed in this order: 37filesystem tests, magic tests, and language tests. 38The 39.Em first 40test that succeeds causes the file type to be printed. 41.Pp 42The type printed will usually contain one of the words 43.Em text 44(the file contains only 45printing characters and a few common control 46characters and is probably safe to read on an 47.Dv ASCII 48terminal), 49.Em executable 50(the file contains the result of compiling a program 51in a form understandable to some 52.Tn UNIX 53kernel or another), 54or 55.Em data 56meaning anything else (data is usually 57.Dq binary 58or non-printable). 59Exceptions are well-known file formats (core files, tar archives) 60that are known to contain binary data. 61When modifying magic files or the program itself, make sure to 62.Em preserve these keywords . 63Users depend on knowing that all the readable files in a directory 64have the word 65.Dq text 66printed. 67Don't do as Berkeley did and change 68.Dq shell commands text 69to 70.Dq shell script . 71.Pp 72The filesystem tests are based on examining the return from a 73.Xr stat 2 74system call. 75The program checks to see if the file is empty, 76or if it's some sort of special file. 77Any known file types appropriate to the system you are running on 78(sockets, symbolic links, or named pipes (FIFOs) on those systems that 79implement them) 80are intuited if they are defined in the system header file 81.In sys/stat.h . 82.Pp 83The magic tests are used to check for files with data in 84particular fixed formats. 85The canonical example of this is a binary executable (compiled program) 86.Dv a.out 87file, whose format is defined in 88.In elf.h , 89.In a.out.h 90and possibly 91.In exec.h 92in the standard include directory. 93These files have a 94.Dq magic number 95stored in a particular place 96near the beginning of the file that tells the 97.Tn UNIX 98operating system 99that the file is a binary executable, and which of several types thereof. 100The concept of a 101.Dq magic number 102has been applied by extension to data files. 103Any file with some invariant identifier at a small fixed 104offset into the file can usually be described in this way. 105The information identifying these files is read from the compiled 106magic file 107.Pa __MAGIC__.mgc , 108or the files in the directory 109.Pa __MAGIC__ 110if the compiled file does not exist. 111In addition, if 112.Pa $HOME/.magic.mgc 113or 114.Pa $HOME/.magic 115exists, it will be used in preference to the system magic files. 116.Pp 117If a file does not match any of the entries in the magic file, 118it is examined to see if it seems to be a text file. 119ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets 120(such as those used on Macintosh and IBM PC systems), 121UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC 122character sets can be distinguished by the different 123ranges and sequences of bytes that constitute printable text 124in each set. 125If a file passes any of these tests, its character set is reported. 126ASCII, ISO-8859-x, UTF-8, and extended-ASCII files are identified 127as 128.Dq text 129because they will be mostly readable on nearly any terminal; 130UTF-16 and EBCDIC are only 131.Dq character data 132because, while 133they contain text, it is text that will require translation 134before it can be read. 135In addition, 136.Nm 137will attempt to determine other characteristics of text-type files. 138If the lines of a file are terminated by CR, CRLF, or NEL, instead 139of the Unix-standard LF, this will be reported. 140Files that contain embedded escape sequences or overstriking 141will also be identified. 142.Pp 143Once 144.Nm 145has determined the character set used in a text-type file, 146it will 147attempt to determine in what language the file is written. 148The language tests look for particular strings (cf. 149.In names.h ) 150that can appear anywhere in the first few blocks of a file. 151For example, the keyword 152.Em .br 153indicates that the file is most likely a 154.Xr troff 1 155input file, just as the keyword 156.Em struct 157indicates a C program. 158These tests are less reliable than the previous 159two groups, so they are performed last. 160The language test routines also test for some miscellany 161(such as 162.Xr tar 1 163archives, JSON files). 164.Pp 165Any file that cannot be identified as having been written 166in any of the character sets listed above is simply said to be 167.Dq data . 168.Sh OPTIONS 169.Bl -tag -width indent 170.It Fl Fl apple 171Causes the 172.Nm 173command to output the file type and creator code as 174used by older MacOS versions. 175The code consists of eight letters, 176the first describing the file type, the latter the creator. 177This option works properly only for file formats that have the 178apple-style output defined. 179.It Fl b , Fl Fl brief 180Do not prepend filenames to output lines (brief mode). 181.It Fl C , Fl Fl compile 182Write a 183.Pa magic.mgc 184output file that contains a pre-parsed version of the magic file or directory. 185.It Fl c , Fl Fl checking-printout 186Cause a checking printout of the parsed form of the magic file. 187This is usually used in conjunction with the 188.Fl m 189option to debug a new magic file before installing it. 190.It Fl d 191Prints internal debugging information to stderr. 192.It Fl E 193On filesystem errors (file not found etc), instead of handling the error 194as regular output as POSIX mandates and keep going, issue an error message 195and exit. 196.It Fl e , Fl Fl exclude Ar testname 197Exclude the test named in 198.Ar testname 199from the list of tests made to determine the file type. 200Valid test names are: 201.Bl -tag -width compress 202.It apptype 203.Dv EMX 204application type (only on EMX). 205.It ascii 206Various types of text files (this test will try to guess the text 207encoding, irrespective of the setting of the 208.Sq encoding 209option). 210.It encoding 211Different text encodings for soft magic tests. 212.It tokens 213Ignored for backwards compatibility. 214.It cdf 215Prints details of Compound Document Files. 216.It compress 217Checks for, and looks inside, compressed files. 218.It csv 219Checks Comma Separated Value files. 220.It elf 221Prints ELF file details, provided soft magic tests are enabled and the 222elf magic is found. 223.It json 224Examines JSON (RFC-7159) files by parsing them for compliance. 225.It soft 226Consults magic files. 227.It tar 228Examines tar files by verifying the checksum of the 512 byte tar header. 229Excluding this test can provide more detailed content description by using 230the soft magic method. 231.It text 232A synonym for 233.Sq ascii . 234.El 235.It Fl Fl exclude-quiet 236Like 237.Fl Fl exclude 238but ignore tests that 239.Nm 240does not know about. 241This is intended for compatibility with older versions of 242.Nm . 243.It Fl Fl extension 244Print a slash-separated list of valid extensions for the file type found. 245.It Fl F , Fl Fl separator Ar separator 246Use the specified string as the separator between the filename and the 247file result returned. 248Defaults to 249.Sq \&: . 250.It Fl f , Fl Fl files-from Ar namefile 251Read the names of the files to be examined from 252.Ar namefile 253(one per line) 254before the argument list. 255Either 256.Ar namefile 257or at least one filename argument must be present; 258to test the standard input, use 259.Sq - 260as a filename argument. 261Please note that 262.Ar namefile 263is unwrapped and the enclosed filenames are processed when this option is 264encountered and before any further options processing is done. 265This allows one to process multiple lists of files with different command line 266arguments on the same 267.Nm 268invocation. 269Thus if you want to set the delimiter, you need to do it before you specify 270the list of files, like: 271.Dq Fl F Ar @ Fl f Ar namefile , 272instead of: 273.Dq Fl f Ar namefile Fl F Ar @ . 274.It Fl h , Fl Fl no-dereference 275This option causes symlinks not to be followed 276(on systems that support symbolic links). 277This is the default if the environment variable 278.Dv POSIXLY_CORRECT 279is not defined. 280.It Fl i , Fl Fl mime 281Causes the 282.Nm 283command to output mime type strings rather than the more 284traditional human readable ones. 285Thus it may say 286.Sq text/plain; charset=us-ascii 287rather than 288.Dq ASCII text . 289.It Fl Fl mime-type , Fl Fl mime-encoding 290Like 291.Fl i , 292but print only the specified element(s). 293.It Fl k , Fl Fl keep-going 294Don't stop at the first match, keep going. 295Subsequent matches will be 296have the string 297.Sq "\[rs]012\- " 298prepended. 299(If you want a newline, see the 300.Fl r 301option.) 302The magic pattern with the highest strength (see the 303.Fl l 304option) comes first. 305.It Fl l , Fl Fl list 306Shows a list of patterns and their strength sorted descending by 307.Xr magic __FSECTION__ 308strength 309which is used for the matching (see also the 310.Fl k 311option). 312.It Fl L , Fl Fl dereference 313This option causes symlinks to be followed, as the like-named option in 314.Xr ls 1 315(on systems that support symbolic links). 316This is the default if the environment variable 317.Ev POSIXLY_CORRECT 318is defined. 319.It Fl m , Fl Fl magic-file Ar magicfiles 320Specify an alternate list of files and directories containing magic. 321This can be a single item, or a colon-separated list. 322If a compiled magic file is found alongside a file or directory, 323it will be used instead. 324.It Fl N , Fl Fl no-pad 325Don't pad filenames so that they align in the output. 326.It Fl n , Fl Fl no-buffer 327Force stdout to be flushed after checking each file. 328This is only useful if checking a list of files. 329It is intended to be used by programs that want filetype output from a pipe. 330.It Fl p , Fl Fl preserve-date 331On systems that support 332.Xr utime 3 333or 334.Xr utimes 2 , 335attempt to preserve the access time of files analyzed, to pretend that 336.Nm 337never read them. 338.It Fl P , Fl Fl parameter Ar name=value 339Set various parameter limits. 340.Bl -column "elf_phnum" "Default" "XXXXXXXXXXXXXXXXXXXXXXXXXXX" -offset indent 341.It Sy "Name" Ta Sy "Default" Ta Sy "Explanation" 342.It Li bytes Ta 1048576 Ta max number of bytes to read from file 343.It Li elf_notes Ta 256 Ta max ELF notes processed 344.It Li elf_phnum Ta 2048 Ta max ELF program sections processed 345.It Li elf_shnum Ta 32768 Ta max ELF sections processed 346.It Li encoding Ta 65536 Ta max number of bytes to scan for encoding evaluation 347.It Li indir Ta 50 Ta recursion limit for indirect magic 348.It Li name Ta 50 Ta use count limit for name/use magic 349.It Li regex Ta 8192 Ta length limit for regex searches 350.El 351.It Fl r , Fl Fl raw 352Don't translate unprintable characters to \eooo. 353Normally 354.Nm 355translates unprintable characters to their octal representation. 356.It Fl s , Fl Fl special-files 357Normally, 358.Nm 359only attempts to read and determine the type of argument files which 360.Xr stat 2 361reports are ordinary files. 362This prevents problems, because reading special files may have peculiar 363consequences. 364Specifying the 365.Fl s 366option causes 367.Nm 368to also read argument files which are block or character special files. 369This is useful for determining the filesystem types of the data in raw 370disk partitions, which are block special files. 371This option also causes 372.Nm 373to disregard the file size as reported by 374.Xr stat 2 375since on some systems it reports a zero size for raw disk partitions. 376.It Fl S , Fl Fl no-sandbox 377On systems where libseccomp 378.Pa ( https://github.com/seccomp/libseccomp ) 379is available, the 380.Fl S 381option disables sandboxing which is enabled by default. 382This option is needed for 383.Nm 384to execute external decompressing programs, 385i.e. when the 386.Fl z 387option is specified and the built-in decompressors are not available. 388On systems where sandboxing is not available, this option has no effect. 389.It Fl v , Fl Fl version 390Print the version of the program and exit. 391.It Fl z , Fl Fl uncompress 392Try to look inside compressed files. 393.It Fl Z , Fl Fl uncompress-noreport 394Try to look inside compressed files, but report information about the contents 395only not the compression. 396.It Fl 0 , Fl Fl print0 397Output a null character 398.Sq \e0 399after the end of the filename. 400Nice to 401.Xr cut 1 402the output. 403This does not affect the separator, which is still printed. 404.Pp 405If this option is repeated more than once, then 406.Nm 407prints just the filename followed by a NUL followed by the description 408(or ERROR: text) followed by a second NUL for each entry. 409.It Fl -help 410Print a help message and exit. 411.El 412.Sh ENVIRONMENT 413The environment variable 414.Ev MAGIC 415can be used to set the default magic file name. 416If that variable is set, then 417.Nm 418will not attempt to open 419.Pa $HOME/.magic . 420.Nm 421adds 422.Dq Pa .mgc 423to the value of this variable as appropriate. 424The environment variable 425.Ev POSIXLY_CORRECT 426controls (on systems that support symbolic links), whether 427.Nm 428will attempt to follow symlinks or not. 429If set, then 430.Nm 431follows symlink, otherwise it does not. 432This is also controlled by the 433.Fl L 434and 435.Fl h 436options. 437.Sh FILES 438.Bl -tag -width __MAGIC__.mgc -compact 439.It Pa __MAGIC__.mgc 440Default compiled list of magic. 441.It Pa __MAGIC__ 442Directory containing default magic files. 443.El 444.Sh EXIT STATUS 445.Nm 446will exit with 447.Dv 0 448if the operation was successful or 449.Dv >0 450if an error was encountered. 451The following errors cause diagnostic messages, but don't affect the program 452exit code (as POSIX requires), unless 453.Fl E 454is specified: 455.Bl -bullet -compact -offset indent 456.It 457A file cannot be found 458.It 459There is no permission to read a file 460.It 461The file type cannot be determined 462.El 463.Sh EXAMPLES 464.Bd -literal -offset indent 465$ file file.c file /dev/{wd0a,hda} 466file.c: C program text 467file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 468 dynamically linked (uses shared libs), stripped 469/dev/wd0a: block special (0/0) 470/dev/hda: block special (3/0) 471 472$ file -s /dev/wd0{b,d} 473/dev/wd0b: data 474/dev/wd0d: x86 boot sector 475 476$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} 477/dev/hda: x86 boot sector 478/dev/hda1: Linux/i386 ext2 filesystem 479/dev/hda2: x86 boot sector 480/dev/hda3: x86 boot sector, extended partition table 481/dev/hda4: Linux/i386 ext2 filesystem 482/dev/hda5: Linux/i386 swap file 483/dev/hda6: Linux/i386 swap file 484/dev/hda7: Linux/i386 swap file 485/dev/hda8: Linux/i386 swap file 486/dev/hda9: empty 487/dev/hda10: empty 488 489$ file -i file.c file /dev/{wd0a,hda} 490file.c: text/x-c 491file: application/x-executable 492/dev/hda: application/x-not-regular-file 493/dev/wd0a: application/x-not-regular-file 494 495.Ed 496.Sh SEE ALSO 497.Xr hexdump 1 , 498.Xr od 1 , 499.Xr strings 1 , 500.Xr magic __FSECTION__ 501.Sh STANDARDS CONFORMANCE 502This program is believed to exceed the System V Interface Definition 503of FILE(CMD), as near as one can determine from the vague language 504contained therein. 505Its behavior is mostly compatible with the System V program of the same name. 506This version knows more magic, however, so it will produce 507different (albeit more accurate) output in many cases. 508.\" URL: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html 509.Pp 510The one significant difference 511between this version and System V 512is that this version treats any white space 513as a delimiter, so that spaces in pattern strings must be escaped. 514For example, 515.Bd -literal -offset indent 516\*[Gt]10 string language impress\ (imPRESS data) 517.Ed 518.Pp 519in an existing magic file would have to be changed to 520.Bd -literal -offset indent 521\*[Gt]10 string language\e impress (imPRESS data) 522.Ed 523.Pp 524In addition, in this version, if a pattern string contains a backslash, 525it must be escaped. 526For example 527.Bd -literal -offset indent 5280 string \ebegindata Andrew Toolkit document 529.Ed 530.Pp 531in an existing magic file would have to be changed to 532.Bd -literal -offset indent 5330 string \e\ebegindata Andrew Toolkit document 534.Ed 535.Pp 536SunOS releases 3.2 and later from Sun Microsystems include a 537.Nm 538command derived from the System V one, but with some extensions. 539This version differs from Sun's only in minor ways. 540It includes the extension of the 541.Sq \*[Am] 542operator, used as, 543for example, 544.Bd -literal -offset indent 545\*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped 546.Ed 547.Sh SECURITY 548On systems where libseccomp 549.Pa ( https://github.com/seccomp/libseccomp ) 550is available, 551.Nm 552is enforces limiting system calls to only the ones necessary for the 553operation of the program. 554This enforcement does not provide any security benefit when 555.Nm 556is asked to decompress input files running external programs with 557the 558.Fl z 559option. 560To enable execution of external decompressors, one needs to disable 561sandboxing using the 562.Fl S 563option. 564.Sh MAGIC DIRECTORY 565The magic file entries have been collected from various sources, 566mainly USENET, and contributed by various authors. 567Christos Zoulas (address below) will collect additional 568or corrected magic file entries. 569A consolidation of magic file entries 570will be distributed periodically. 571.Pp 572The order of entries in the magic file is significant. 573Depending on what system you are using, the order that 574they are put together may be incorrect. 575If your old 576.Nm 577command uses a magic file, 578keep the old magic file around for comparison purposes 579(rename it to 580.Pa __MAGIC__.orig ) . 581.Sh HISTORY 582There has been a 583.Nm 584command in every 585.Dv UNIX since at least Research Version 4 586(man page dated November, 1973). 587The System V version introduced one significant major change: 588the external list of magic types. 589This slowed the program down slightly but made it a lot more flexible. 590.Pp 591This program, based on the System V version, 592was written by Ian Darwin 593.Aq ian@darwinsys.com 594without looking at anybody else's source code. 595.Pp 596John Gilmore revised the code extensively, making it better than 597the first version. 598Geoff Collyer found several inadequacies 599and provided some magic file entries. 600Contributions of the 601.Sq \*[Am] 602operator by Rob McMahon, 603.Aq cudcv@warwick.ac.uk , 6041989. 605.Pp 606Guy Harris, 607.Aq guy@netapp.com , 608made many changes from 1993 to the present. 609.Pp 610Primary development and maintenance from 1990 to the present by 611Christos Zoulas 612.Aq christos@astron.com . 613.Pp 614Altered by Chris Lowth 615.Aq chris@lowth.com , 6162000: handle the 617.Fl i 618option to output mime type strings, using an alternative 619magic file and internal logic. 620.Pp 621Altered by Eric Fischer 622.Aq enf@pobox.com , 623July, 2000, 624to identify character codes and attempt to identify the languages 625of non-ASCII files. 626.Pp 627Altered by Reuben Thomas 628.Aq rrt@sc3d.org , 6292007-2011, to improve MIME support, merge MIME and non-MIME magic, 630support directories as well as files of magic, apply many bug fixes, 631update and fix a lot of magic, improve the build system, improve the 632documentation, and rewrite the Python bindings in pure Python. 633.Pp 634The list of contributors to the 635.Sq magic 636directory (magic files) 637is too long to include here. 638You know who you are; thank you. 639Many contributors are listed in the source files. 640.Sh LEGAL NOTICE 641Copyright (c) Ian F. Darwin, Toronto, Canada, 1986-1999. 642Covered by the standard Berkeley Software Distribution copyright; see the file 643COPYING in the source distribution. 644.Pp 645The files 646.Pa tar.h 647and 648.Pa is_tar.c 649were written by John Gilmore from his public-domain 650.Xr tar 1 651program, and are not covered by the above license. 652.Sh BUGS 653Please report bugs and send patches to the bug tracker at 654.Pa https://bugs.astron.com/ 655or the mailing list at 656.Aq file@astron.com 657(visit 658.Pa https://mailman.astron.com/mailman/listinfo/file 659first to subscribe). 660.Sh TODO 661Fix output so that tests for MIME and APPLE flags are not needed all 662over the place, and actual output is only done in one place. 663This needs a design. 664Suggestion: push possible outputs on to a list, then pick the 665last-pushed (most specific, one hopes) value at the end, or 666use a default if the list is empty. 667This should not slow down evaluation. 668.Pp 669The handling of 670.Dv MAGIC_CONTINUE 671and printing \e012- between entries is clumsy and complicated; refactor 672and centralize. 673.Pp 674Some of the encoding logic is hard-coded in encoding.c and can be moved 675to the magic files if we had a !:charset annotation. 676.Pp 677Continue to squash all magic bugs. 678See Debian BTS for a good source. 679.Pp 680Store arbitrarily long strings, for example for %s patterns, so that 681they can be printed out. 682Fixes Debian bug #271672. 683This can be done by allocating strings in a string pool, storing the 684string pool at the end of the magic file and converting all the string 685pointers to relative offsets from the string pool. 686.Pp 687Add syntax for relative offsets after current level (Debian bug #466037). 688.Pp 689Make file -ki work, i.e. give multiple MIME types. 690.Pp 691Add a zip library so we can peek inside Office2007 documents to 692print more details about their contents. 693.Pp 694Add an option to print URLs for the sources of the file descriptions. 695.Pp 696Combine script searches and add a way to map executable names to MIME 697types (e.g. have a magic value for !:mime which causes the resulting 698string to be looked up in a table). 699This would avoid adding the same magic repeatedly for each new 700hash-bang interpreter. 701.Pp 702When a file descriptor is available, we can skip and adjust the buffer 703instead of the hacky buffer management we do now. 704.Pp 705Fix 706.Dq name 707and 708.Dq use 709to check for consistency at compile time (duplicate 710.Dq name , 711.Dq use 712pointing to undefined 713.Dq name 714). 715Make 716.Dq name 717/ 718.Dq use 719more efficient by keeping a sorted list of names. 720Special-case ^ to flip endianness in the parser so that it does not 721have to be escaped, and document it. 722.Pp 723If the offsets specified internally in the file exceed the buffer size 724( 725.Dv HOWMANY 726variable in file.h), then we don't seek to that offset, but we give up. 727It would be better if buffer managements was done when the file descriptor 728is available so we can seek around the file. 729One must be careful though because this has performance and thus security 730considerations, because one can slow down things by repeateadly seeking. 731.Pp 732There is support now for keeping separate buffers and having offsets from 733the end of the file, but the internal buffer management still needs an 734overhaul. 735.Sh AVAILABILITY 736You can obtain the original author's latest version by anonymous FTP 737on 738.Pa ftp.astron.com 739in the directory 740.Pa /pub/file/file-X.YZ.tar.gz . 741