1.\" $File: magic.man,v 1.81 2014/03/08 17:28:08 christos Exp $ 2.Dd April 22, 2013 3.Dt MAGIC __FSECTION__ 4.Os 5.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 6.Sh NAME 7.Nm magic 8.Nd file command's magic pattern file 9.Sh DESCRIPTION 10This manual page documents the format of the magic file as 11used by the 12.Xr file __CSECTION__ 13command, version __VERSION__. 14The 15.Xr file __CSECTION__ 16command identifies the type of a file using, 17among other tests, 18a test for whether the file contains certain 19.Dq "magic patterns" . 20The file 21.Pa __MAGIC__ 22specifies what patterns are to be tested for, what message or 23MIME type to print if a particular pattern is found, 24and additional information to extract from the file. 25.Pp 26Each line of the file specifies a test to be performed. 27A test compares the data starting at a particular offset 28in the file with a byte value, a string or a numeric value. 29If the test succeeds, a message is printed. 30The line consists of the following fields: 31.Bl -tag -width ".Dv message" 32.It Dv offset 33A number specifying the offset, in bytes, into the file of the data 34which is to be tested. 35.It Dv type 36The type of the data to be tested. 37The possible values are: 38.Bl -tag -width ".Dv lestring16" 39.It Dv byte 40A one-byte value. 41.It Dv short 42A two-byte value in this machine's native byte order. 43.It Dv long 44A four-byte value in this machine's native byte order. 45.It Dv quad 46An eight-byte value in this machine's native byte order. 47.It Dv float 48A 32-bit single precision IEEE floating point number in this machine's native byte order. 49.It Dv double 50A 64-bit double precision IEEE floating point number in this machine's native byte order. 51.It Dv string 52A string of bytes. 53The string type specification can be optionally followed 54by /[WwcCtbT]*. 55The 56.Dq W 57flag compacts whitespace in the target, which must 58contain at least one whitespace character. 59If the magic has 60.Dv n 61consecutive blanks, the target needs at least 62.Dv n 63consecutive blanks to match. 64The 65.Dq w 66flag treats every blank in the magic as an optional blank. 67The 68.Dq c 69flag specifies case insensitive matching: lower case 70characters in the magic match both lower and upper case characters in the 71target, whereas upper case characters in the magic only match upper case 72characters in the target. 73The 74.Dq C 75flag specifies case insensitive matching: upper case 76characters in the magic match both lower and upper case characters in the 77target, whereas lower case characters in the magic only match upper case 78characters in the target. 79To do a complete case insensitive match, specify both 80.Dq c 81and 82.Dq C . 83The 84.Dq t 85flag forces the test to be done for text files, while the 86.Dq b 87flag forces the test to be done for binary files. 88The 89.Dq T 90flag causes the string to be trimmed, i.e. leading and trailing whitespace 91is deleted before the string is printed. 92.It Dv pstring 93A Pascal-style string where the first byte/short/int is interpreted as the 94unsigned length. 95The length defaults to byte and can be specified as a modifier. 96The following modifiers are supported: 97.Bl -tag -compact -width B 98.It B 99A byte length (default). 100.It H 101A 2 byte big endian length. 102.It h 103A 2 byte big little length. 104.It L 105A 4 byte big endian length. 106.It l 107A 4 byte big little length. 108.It J 109The length includes itself in its count. 110.El 111The string is not NUL terminated. 112.Dq J 113is used rather than the more 114valuable 115.Dq I 116because this type of length is a feature of the JPEG 117format. 118.It Dv date 119A four-byte value interpreted as a UNIX date. 120.It Dv qdate 121A eight-byte value interpreted as a UNIX date. 122.It Dv ldate 123A four-byte value interpreted as a UNIX-style date, but interpreted as 124local time rather than UTC. 125.It Dv qldate 126An eight-byte value interpreted as a UNIX-style date, but interpreted as 127local time rather than UTC. 128.It Dv qwdate 129An eight-byte value interpreted as a Windows-style date. 130.It Dv beid3 131A 32-bit ID3 length in big-endian byte order. 132.It Dv beshort 133A two-byte value in big-endian byte order. 134.It Dv belong 135A four-byte value in big-endian byte order. 136.It Dv bequad 137An eight-byte value in big-endian byte order. 138.It Dv befloat 139A 32-bit single precision IEEE floating point number in big-endian byte order. 140.It Dv bedouble 141A 64-bit double precision IEEE floating point number in big-endian byte order. 142.It Dv bedate 143A four-byte value in big-endian byte order, 144interpreted as a Unix date. 145.It Dv beqdate 146An eight-byte value in big-endian byte order, 147interpreted as a Unix date. 148.It Dv beldate 149A four-byte value in big-endian byte order, 150interpreted as a UNIX-style date, but interpreted as local time rather 151than UTC. 152.It Dv beqldate 153An eight-byte value in big-endian byte order, 154interpreted as a UNIX-style date, but interpreted as local time rather 155than UTC. 156.It Dv beqwdate 157An eight-byte value in big-endian byte order, 158interpreted as a Windows-style date. 159.It Dv bestring16 160A two-byte unicode (UCS16) string in big-endian byte order. 161.It Dv leid3 162A 32-bit ID3 length in little-endian byte order. 163.It Dv leshort 164A two-byte value in little-endian byte order. 165.It Dv lelong 166A four-byte value in little-endian byte order. 167.It Dv lequad 168An eight-byte value in little-endian byte order. 169.It Dv lefloat 170A 32-bit single precision IEEE floating point number in little-endian byte order. 171.It Dv ledouble 172A 64-bit double precision IEEE floating point number in little-endian byte order. 173.It Dv ledate 174A four-byte value in little-endian byte order, 175interpreted as a UNIX date. 176.It Dv leqdate 177An eight-byte value in little-endian byte order, 178interpreted as a UNIX date. 179.It Dv leldate 180A four-byte value in little-endian byte order, 181interpreted as a UNIX-style date, but interpreted as local time rather 182than UTC. 183.It Dv leqldate 184An eight-byte value in little-endian byte order, 185interpreted as a UNIX-style date, but interpreted as local time rather 186than UTC. 187.It Dv leqwdate 188An eight-byte value in little-endian byte order, 189interpreted as a Windows-style date. 190.It Dv lestring16 191A two-byte unicode (UCS16) string in little-endian byte order. 192.It Dv melong 193A four-byte value in middle-endian (PDP-11) byte order. 194.It Dv medate 195A four-byte value in middle-endian (PDP-11) byte order, 196interpreted as a UNIX date. 197.It Dv meldate 198A four-byte value in middle-endian (PDP-11) byte order, 199interpreted as a UNIX-style date, but interpreted as local time rather 200than UTC. 201.It Dv indirect 202Starting at the given offset, consult the magic database again. 203.It Dv name 204Define a 205.Dq named 206magic instance that can be called from another 207.Dv use 208magic entry, like a subroutine call. 209Named instance direct magic offsets are relative to the offset of the 210previous matched entry, but indirect offsets are relative to the beginning 211of the file as usual. 212Named magic entries always match. 213.It Dv use 214Recursively call the named magic starting from the current offset. 215If the name of the referenced begins with a 216.Dv ^ 217then the endianness of the magic is switched; if the magic mentioned 218.Dv leshort 219for example, 220it is treated as 221.Dv beshort 222and vice versa. 223This is useful to avoid duplicating the rules for different endianness. 224.It Dv regex 225A regular expression match in extended POSIX regular expression syntax 226(like egrep). 227Regular expressions can take exponential time to process, and their 228performance is hard to predict, so their use is discouraged. 229When used in production environments, their performance 230should be carefully checked. 231The type specification can be optionally followed by 232.Dv /[c][s] . 233The 234.Dq c 235flag makes the match case insensitive, while the 236.Dq s 237flag update the offset to the start offset of the match, rather than the end. 238The regular expression is tested against line 239.Dv N + 1 240onwards, where 241.Dv N 242is the given offset. 243Line endings are assumed to be in the machine's native format. 244.Dv ^ 245and 246.Dv $ 247match the beginning and end of individual lines, respectively, 248not beginning and end of file. 249.It Dv search 250A literal string search starting at the given offset. 251The same modifier flags can be used as for string patterns. 252The search expression must contain the range in the form 253.Dv /number, 254that is the number of positions at which the match will be 255attempted, starting from the start offset. 256This is suitable for 257searching larger binary expressions with variable offsets, using 258.Dv \e 259escapes for special characters. 260The order of modifier and number is not relevant. 261.It Dv default 262This is intended to be used with the test 263.Em x 264(which is always true) and it has no type. 265It matches when no other test at that continuation level has matched before. 266Clearing that matched tests for a continuation level, can be done using the 267.Dv clear 268test. 269.It Dv clear 270This test is always true and clears the match flag for that continuation level. 271It is intended to be used with the 272.Dv default 273test. 274.El 275.Pp 276For compatibility with the Single 277.Ux 278Standard, the type specifiers 279.Dv dC 280and 281.Dv d1 282are equivalent to 283.Dv byte , 284the type specifiers 285.Dv uC 286and 287.Dv u1 288are equivalent to 289.Dv ubyte , 290the type specifiers 291.Dv dS 292and 293.Dv d2 294are equivalent to 295.Dv short , 296the type specifiers 297.Dv uS 298and 299.Dv u2 300are equivalent to 301.Dv ushort , 302the type specifiers 303.Dv dI , 304.Dv dL , 305and 306.Dv d4 307are equivalent to 308.Dv long , 309the type specifiers 310.Dv uI , 311.Dv uL , 312and 313.Dv u4 314are equivalent to 315.Dv ulong , 316the type specifier 317.Dv d8 318is equivalent to 319.Dv quad , 320the type specifier 321.Dv u8 322is equivalent to 323.Dv uquad , 324and the type specifier 325.Dv s 326is equivalent to 327.Dv string . 328In addition, the type specifier 329.Dv dQ 330is equivalent to 331.Dv quad 332and the type specifier 333.Dv uQ 334is equivalent to 335.Dv uquad . 336.Pp 337Each top-level magic pattern (see below for an explanation of levels) 338is classified as text or binary according to the types used. 339Types 340.Dq regex 341and 342.Dq search 343are classified as text tests, unless non-printable characters are used 344in the pattern. 345All other tests are classified as binary. 346A top-level 347pattern is considered to be a test text when all its patterns are text 348patterns; otherwise, it is considered to be a binary pattern. 349When 350matching a file, binary patterns are tried first; if no match is 351found, and the file looks like text, then its encoding is determined 352and the text patterns are tried. 353.Pp 354The numeric types may optionally be followed by 355.Dv \*[Am] 356and a numeric value, 357to specify that the value is to be AND'ed with the 358numeric value before any comparisons are done. 359Prepending a 360.Dv u 361to the type indicates that ordered comparisons should be unsigned. 362.It Dv test 363The value to be compared with the value from the file. 364If the type is 365numeric, this value 366is specified in C form; if it is a string, it is specified as a C string 367with the usual escapes permitted (e.g. \en for new-line). 368.Pp 369Numeric values 370may be preceded by a character indicating the operation to be performed. 371It may be 372.Dv = , 373to specify that the value from the file must equal the specified value, 374.Dv \*[Lt] , 375to specify that the value from the file must be less than the specified 376value, 377.Dv \*[Gt] , 378to specify that the value from the file must be greater than the specified 379value, 380.Dv \*[Am] , 381to specify that the value from the file must have set all of the bits 382that are set in the specified value, 383.Dv ^ , 384to specify that the value from the file must have clear any of the bits 385that are set in the specified value, or 386.Dv ~ , 387the value specified after is negated before tested. 388.Dv x , 389to specify that any value will match. 390If the character is omitted, it is assumed to be 391.Dv = . 392Operators 393.Dv \*[Am] , 394.Dv ^ , 395and 396.Dv ~ 397don't work with floats and doubles. 398The operator 399.Dv !\& 400specifies that the line matches if the test does 401.Em not 402succeed. 403.Pp 404Numeric values are specified in C form; e.g. 405.Dv 13 406is decimal, 407.Dv 013 408is octal, and 409.Dv 0x13 410is hexadecimal. 411.Pp 412For string values, the string from the 413file must match the specified string. 414The operators 415.Dv = , 416.Dv \*[Lt] 417and 418.Dv \*[Gt] 419(but not 420.Dv \*[Am] ) 421can be applied to strings. 422The length used for matching is that of the string argument 423in the magic file. 424This means that a line can match any non-empty string (usually used to 425then print the string), with 426.Em \*[Gt]\e0 427(because all non-empty strings are greater than the empty string). 428.Pp 429Dates are treated as numerical values in the respective internal 430representation. 431.Pp 432The special test 433.Em x 434always evaluates to true. 435.It Dv message 436The message to be printed if the comparison succeeds. 437If the string contains a 438.Xr printf 3 439format specification, the value from the file (with any specified masking 440performed) is printed using the message as the format string. 441If the string begins with 442.Dq \eb , 443the message printed is the remainder of the string with no whitespace 444added before it: multiple matches are normally separated by a single 445space. 446.El 447.Pp 448An APPLE 4+4 character APPLE creator and type can be specified as: 449.Bd -literal -offset indent 450!:apple CREATYPE 451.Ed 452.Pp 453A MIME type is given on a separate line, which must be the next 454non-blank or comment line after the magic line that identifies the 455file type, and has the following format: 456.Bd -literal -offset indent 457!:mime MIMETYPE 458.Ed 459.Pp 460i.e. the literal string 461.Dq !:mime 462followed by the MIME type. 463.Pp 464An optional strength can be supplied on a separate line which refers to 465the current magic description using the following format: 466.Bd -literal -offset indent 467!:strength OP VALUE 468.Ed 469.Pp 470The operand 471.Dv OP 472can be: 473.Dv + , 474.Dv - , 475.Dv * , 476or 477.Dv / 478and 479.Dv VALUE 480is a constant between 0 and 255. 481This constant is applied using the specified operand 482to the currently computed default magic strength. 483.Pp 484Some file formats contain additional information which is to be printed 485along with the file type or need additional tests to determine the true 486file type. 487These additional tests are introduced by one or more 488.Em \*[Gt] 489characters preceding the offset. 490The number of 491.Em \*[Gt] 492on the line indicates the level of the test; a line with no 493.Em \*[Gt] 494at the beginning is considered to be at level 0. 495Tests are arranged in a tree-like hierarchy: 496if the test on a line at level 497.Em n 498succeeds, all following tests at level 499.Em n+1 500are performed, and the messages printed if the tests succeed, until a line 501with level 502.Em n 503(or less) appears. 504For more complex files, one can use empty messages to get just the 505"if/then" effect, in the following way: 506.Bd -literal -offset indent 5070 string MZ 508\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 509\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 510.Ed 511.Pp 512Offsets do not need to be constant, but can also be read from the file 513being examined. 514If the first character following the last 515.Em \*[Gt] 516is a 517.Em \&( 518then the string after the parenthesis is interpreted as an indirect offset. 519That means that the number after the parenthesis is used as an offset in 520the file. 521The value at that offset is read, and is used again as an offset 522in the file. 523Indirect offsets are of the form: 524.Em (( x [.[bislBISL]][+\-][ y ]) . 525The value of 526.Em x 527is used as an offset in the file. 528A byte, id3 length, short or long is read at that offset depending on the 529.Em [bislBISLm] 530type specifier. 531The capitalized types interpret the number as a big endian 532value, whereas the small letter versions interpret the number as a little 533endian value; 534the 535.Em m 536type interprets the number as a middle endian (PDP-11) value. 537To that number the value of 538.Em y 539is added and the result is used as an offset in the file. 540The default type if one is not specified is long. 541.Pp 542That way variable length structures can be examined: 543.Bd -literal -offset indent 544# MS Windows executables are also valid MS-DOS executables 5450 string MZ 546\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 547# skip the whole block below if it is not an extended executable 548\*[Gt]0x18 leshort \*[Gt]0x3f 549\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 550\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 551.Ed 552.Pp 553This strategy of examining has a drawback: You must make sure that 554you eventually print something, or users may get empty output (like, when 555there is neither PE\e0\e0 nor LE\e0\e0 in the above example) 556.Pp 557If this indirect offset cannot be used directly, simple calculations are 558possible: appending 559.Em [+-*/%\*[Am]|^]number 560inside parentheses allows one to modify 561the value read from the file before it is used as an offset: 562.Bd -literal -offset indent 563# MS Windows executables are also valid MS-DOS executables 5640 string MZ 565# sometimes, the value at 0x18 is less that 0x40 but there's still an 566# extended executable, simply appended to the file 567\*[Gt]0x18 leshort \*[Lt]0x40 568\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 569\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 570.Ed 571.Pp 572Sometimes you do not know the exact offset as this depends on the length or 573position (when indirection was used before) of preceding fields. 574You can specify an offset relative to the end of the last up-level 575field using 576.Sq \*[Am] 577as a prefix to the offset: 578.Bd -literal -offset indent 5790 string MZ 580\*[Gt]0x18 leshort \*[Gt]0x3f 581\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 582# immediately following the PE signature is the CPU type 583\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 584\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 585.Ed 586.Pp 587Indirect and relative offsets can be combined: 588.Bd -literal -offset indent 5890 string MZ 590\*[Gt]0x18 leshort \*[Lt]0x40 591\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 592# if it's not COFF, go back 512 bytes and add the offset taken 593# from byte 2/3, which is yet another way of finding the start 594# of the extended executable 595\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 596.Ed 597.Pp 598Or the other way around: 599.Bd -literal -offset indent 6000 string MZ 601\*[Gt]0x18 leshort \*[Gt]0x3f 602\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 603# at offset 0x80 (-4, since relative offsets start at the end 604# of the up-level match) inside the LE header, we find the absolute 605# offset to the code area, where we look for a specific signature 606\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 607.Ed 608.Pp 609Or even both! 610.Bd -literal -offset indent 6110 string MZ 612\*[Gt]0x18 leshort \*[Gt]0x3f 613\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 614# at offset 0x58 inside the LE header, we find the relative offset 615# to a data area where we look for a specific signature 616\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 617.Ed 618.Pp 619If you have to deal with offset/length pairs in your file, even the 620second value in a parenthesized expression can be taken from the file itself, 621using another set of parentheses. 622Note that this additional indirect offset is always relative to the 623start of the main indirect offset. 624.Bd -literal -offset indent 6250 string MZ 626\*[Gt]0x18 leshort \*[Gt]0x3f 627\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 628# search for the PE section called ".idata"... 629\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 630# ...and go to the end of it, calculated from start+length; 631# these are located 14 and 10 bytes after the section name 632\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 633.Ed 634.Pp 635If you have a list of known avalues at a particular continuation level, 636and you want to provide a switch-like default case: 637.Bd -literal -offset indent 638# clear that continuation level match 639\*[Gt]18 clear 640\*[Gt]18 lelong 1 one 641\*[Gt]18 lelong 2 two 642\*[Gt]18 default x 643# print default match 644\*[Gt]\*[Gt]18 lelong x unmatched 0x%x 645.Ed 646.Sh SEE ALSO 647.Xr file __CSECTION__ 648\- the command that reads this file. 649.Sh BUGS 650The formats 651.Dv long , 652.Dv belong , 653.Dv lelong , 654.Dv melong , 655.Dv short , 656.Dv beshort , 657and 658.Dv leshort 659do not depend on the length of the C data types 660.Dv short 661and 662.Dv long 663on the platform, even though the Single 664.Ux 665Specification implies that they do. However, as OS X Mountain Lion has 666passed the Single 667.Ux 668Specification validation suite, and supplies a version of 669.Xr file __CSECTION__ 670in which they do not depend on the sizes of the C data types and that is 671built for a 64-bit environment in which 672.Dv long 673is 8 bytes rather than 4 bytes, presumably the validation suite does not 674test whether, for example 675.Dv long 676refers to an item with the same size as the C data type 677.Dv long . 678There should probably be 679.Dv type 680names 681.Dv int8 , 682.Dv uint8 , 683.Dv int16 , 684.Dv uint16 , 685.Dv int32 , 686.Dv uint32 , 687.Dv int64 , 688and 689.Dv uint64 , 690and specified-byte-order variants of them, 691to make it clearer that those types have specified widths. 692.\" 693.\" From: guy@sun.uucp (Guy Harris) 694.\" Newsgroups: net.bugs.usg 695.\" Subject: /etc/magic's format isn't well documented 696.\" Message-ID: <2752@sun.uucp> 697.\" Date: 3 Sep 85 08:19:07 GMT 698.\" Organization: Sun Microsystems, Inc. 699.\" Lines: 136 700.\" 701.\" Here's a manual page for the format accepted by the "file" made by adding 702.\" the changes I posted to the S5R2 version. 703.\" 704.\" Modified for Ian Darwin's version of the file command. 705