@c PSPP - a program for statistical analysis. @c Copyright (C) 2019 Free Software Foundation, Inc. @c Permission is granted to copy, distribute and/or modify this document @c under the terms of the GNU Free Documentation License, Version 1.3 @c or any later version published by the Free Software Foundation; @c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. @c A copy of the license is included in the section entitled "GNU @c Free Documentation License". @c @node System File Format @appendix System File Format A system file encapsulates a set of cases and dictionary information that describes how they may be interpreted. This chapter describes the format of a system file. System files use four data types: 8-bit characters, 32-bit integers, 64-bit integers, and 64-bit floating points, called here @code{char}, @code{int32}, @code{int64}, and @code{flt64}, respectively. Data is not necessarily aligned on a word or double-word boundary: the long variable name record (@pxref{Long Variable Names Record}) and very long string records (@pxref{Very Long String Record}) have arbitrary byte length and can therefore cause all data coming after them in the file to be misaligned. Integer data in system files may be big-endian or little-endian. A reader may detect the endianness of a system file by examining @code{layout_code} in the file header record (@pxref{layout_code,,@code{layout_code}}). Floating-point data in system files may nominally be in IEEE 754, IBM, or VAX formats. A reader may detect the floating-point format in use by examining @code{bias} in the file header record (@pxref{bias,,@code{bias}}). PSPP detects big-endian and little-endian integer formats in system files and translates as necessary. PSPP also detects the floating-point format in use, as well as the endianness of IEEE 754 floating-point numbers, and translates as needed. However, only IEEE 754 numbers with the same endianness as integer data in the same file have actually been observed in system files, and it is likely that other formats are obsolete or were never used. System files use a few floating point values for special purposes: @table @asis @item SYSMIS The system-missing value is represented by the largest possible negative number in the floating point format (@code{-DBL_MAX}). @item HIGHEST HIGHEST is used as the high end of a missing value range with an unbounded maximum. It is represented by the largest possible positive number (@code{DBL_MAX}). @item LOWEST LOWEST is used as the low end of a missing value range with an unbounded minimum. It was originally represented by the second-largest negative number (in IEEE 754 format, @code{0xffeffffffffffffe}). System files written by SPSS 21 and later instead use the largest negative number (@code{-DBL_MAX}), the same value as SYSMIS. This does not lead to ambiguity because LOWEST appears in system files only in missing value ranges, which never contain SYSMIS. @end table System files may use most character encodings based on an 8-bit unit. UTF-16 and UTF-32, based on wider units, appear to be unacceptable. @code{rec_type} in the file header record is sufficient to distinguish between ASCII and EBCDIC based encodings. The best way to determine the specific encoding in use is to consult the character encoding record (@pxref{Character Encoding Record}), if present, and failing that the @code{character_code} in the machine integer info record (@pxref{Machine Integer Info Record}). The same encoding should be used for the dictionary and the data in the file, although it is possible to artificially synthesize files that use different encodings (@pxref{Character Encoding Record}). @menu * System File Record Structure:: * File Header Record:: * Variable Record:: * Value Labels Records:: * Document Record:: * Machine Integer Info Record:: * Machine Floating-Point Info Record:: * Multiple Response Sets Records:: * Extra Product Info Record:: * Variable Display Parameter Record:: * Long Variable Names Record:: * Very Long String Record:: * Character Encoding Record:: * Long String Value Labels Record:: * Long String Missing Values Record:: * Data File and Variable Attributes Records:: * Extended Number of Cases Record:: * Other Informational Records:: * Dictionary Termination Record:: * Data Record:: @end menu @node System File Record Structure @section System File Record Structure System files are divided into records with the following format: @example int32 type; char data[]; @end example This header does not identify the length of the @code{data} or any information about what it contains, so the system file reader must understand the format of @code{data} based on @code{type}. However, records with type 7, called @dfn{extension records}, have a stricter format: @example int32 type; int32 subtype; int32 size; int32 count; char data[size * count]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. This value identifies a particular kind of extension record. @item int32 size; The size of each piece of data that follows the header, in bytes. Known extension records use 1, 4, or 8, for @code{char}, @code{int32}, and @code{flt64} format data, respectively. @item int32 count; The number of pieces of data that follow the header. @item char data[size * count]; Data, whose format and interpretation depend on the subtype. @end table An extension record contains exactly @code{size * count} bytes of data, which allows a reader that does not understand an extension record to skip it. Extension records provide only nonessential information, so this allows for files written by newer software to preserve backward compatibility with older or less capable readers. Records in a system file must appear in the following order: @itemize @bullet @item File header record. @item Variable records. @item All pairs of value labels records and value label variables records, if present. @item Document record, if present. @item Extension (type 7) records, in ascending numerical order of their subtypes. System files written by SPSS include at most one of each kind of extension record. This is generally true of system files written by other software as well, with known exceptions noted below in the individual sections about each type of record. @item Dictionary termination record. @item Data record. @end itemize We advise authors of programs that read system files to tolerate format variations. Various kinds of misformatting and corruption have been observed in system files written by SPSS and other software alike. In particular, because extension records provide nonessential information, it is generally better to ignore an extension record entirely than to refuse to read a system file. The following sections describe the known kinds of records. @node File Header Record @section File Header Record A system file begins with the file header, with the following format: @example char rec_type[4]; char prod_name[60]; int32 layout_code; int32 nominal_case_size; int32 compression; int32 weight_index; int32 ncases; flt64 bias; char creation_date[9]; char creation_time[8]; char file_label[64]; char padding[3]; @end example @table @code @item char rec_type[4]; Record type code, either @samp{$FL2} for system files with uncompressed data or data compressed with simple bytecode compression, or @samp{$FL3} for system files with ZLIB compressed data. This is truly a character field that uses the character encoding as other strings. Thus, in a file with an ASCII-based character encoding this field contains @code{24 46 4c 32} or @code{24 46 4c 33}, and in a file with an EBCDIC-based encoding this field contains @code{5b c6 d3 f2}. (No EBCDIC-based ZLIB-compressed files have been observed.) @item char prod_name[60]; Product identification string. This always begins with the characters @samp{@@(#) SPSS DATA FILE}. PSPP uses the remaining characters to give its version and the operating system name; for example, @samp{GNU pspp 0.1.4 - sparc-sun-solaris2.5.2}. The string is truncated if it would be longer than 60 characters; otherwise it is padded on the right with spaces. The product name field allow readers to behave differently based on quirks in the way that particular software writes system files. @xref{Value Labels Records}, for the detail of the quirk that the PSPP system file reader tolerates in files written by ReadStat, which has @code{https://github.com/WizardMac/ReadStat} in @code{prod_name}. @anchor{layout_code} @item int32 layout_code; Normally set to 2, although a few system files have been spotted in the wild with a value of 3 here. PSPP use this value to determine the file's integer endianness (@pxref{System File Format}). @item int32 nominal_case_size; Number of data elements per case. This is the number of variables, except that long string variables add extra data elements (one for every 8 characters after the first 8). However, string variables do not contribute to this value beyond the first 255 bytes. Further, some software always writes -1 or 0 in this field. In general, it is unsafe for systems reading system files to rely upon this value. @item int32 compression; Set to 0 if the data in the file is not compressed, 1 if the data is compressed with simple bytecode compression, 2 if the data is ZLIB compressed. This field has value 2 if and only if @code{rec_type} is @samp{$FL3}. @item int32 weight_index; If one of the variables in the data set is used as a weighting variable, set to the dictionary index of that variable, plus 1 (@pxref{Dictionary Index}). Otherwise, set to 0. @item int32 ncases; Set to the number of cases in the file if it is known, or -1 otherwise. In the general case it is not possible to determine the number of cases that will be output to a system file at the time that the header is written. The way that this is dealt with is by writing the entire system file, including the header, then seeking back to the beginning of the file and writing just the @code{ncases} field. For files in which this is not valid, the seek operation fails. In this case, @code{ncases} remains -1. @anchor{bias} @item flt64 bias; Compression bias, ordinarily set to 100. Only integers between @code{1 - bias} and @code{251 - bias} can be compressed. By assuming that its value is 100, PSPP uses @code{bias} to determine the file's floating-point format and endianness (@pxref{System File Format}). If the compression bias is not 100, PSPP cannot auto-detect the floating-point format and assumes that it is IEEE 754 format with the same endianness as the system file's integers, which is correct for all known system files. @item char creation_date[9]; Date of creation of the system file, in @samp{dd mmm yy} format, with the month as standard English abbreviations, using an initial capital letter and following with lowercase. If the date is not available then this field is arbitrarily set to @samp{01 Jan 70}. @item char creation_time[8]; Time of creation of the system file, in @samp{hh:mm:ss} format and using 24-hour time. If the time is not available then this field is arbitrarily set to @samp{00:00:00}. @item char file_label[64]; File label declared by the user, if any (@pxref{FILE LABEL,,,pspp, PSPP Users Guide}). Padded on the right with spaces. A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the more usual LF-only or CR LF line ends. @item char padding[3]; Ignored padding bytes to make the structure a multiple of 32 bits in length. Set to zeros. @end table @node Variable Record @section Variable Record There must be one variable record for each numeric variable and each string variable with width 8 bytes or less. String variables wider than 8 bytes have one variable record for each 8 bytes, rounding up. The first variable record for a long string specifies the variable's correct dictionary information. Subsequent variable records for a long string are filled with dummy information: a type of -1, no variable label or missing values, print and write formats that are ignored, and an empty string as name. A few system files have been encountered that include a variable label on dummy variable records, so readers should take care to parse dummy variable records in the same way as other variable records. @anchor{Dictionary Index} The @dfn{dictionary index} of a variable is a 1-based offset in the set of variable records, including dummy variable records for long string variables. The first variable record has a dictionary index of 1, the second has a dictionary index of 2, and so on. The system file format does not directly support string variables wider than 255 bytes. Such very long string variables are represented by a number of narrower string variables. @xref{Very Long String Record}, for details. A system file should contain at least one variable and thus at least one variable record, but system files have been observed in the wild without any variables (thus, no data either). @example int32 rec_type; int32 type; int32 has_var_label; int32 n_missing_values; int32 print; int32 write; char name[8]; /* @r{Present only if @code{has_var_label} is 1.} */ int32 label_len; char label[]; /* @r{Present only if @code{n_missing_values} is nonzero}. */ flt64 missing_values[]; @end example @table @code @item int32 rec_type; Record type code. Always set to 2. @item int32 type; Variable type code. Set to 0 for a numeric variable. For a short string variable or the first part of a long string variable, this is set to the width of the string. For the second and subsequent parts of a long string variable, set to -1, and the remaining fields in the structure are ignored. @item int32 has_var_label; If this variable has a variable label, set to 1; otherwise, set to 0. @item int32 n_missing_values; If the variable has no missing values, set to 0. If the variable has one, two, or three discrete missing values, set to 1, 2, or 3, respectively. If the variable has a range for missing variables, set to -2; if the variable has a range for missing variables plus a single discrete value, set to -3. A long string variable always has the value 0 here. A separate record indicates missing values for long string variables (@pxref{Long String Missing Values Record}). @item int32 print; Print format for this variable. See below. @item int32 write; Write format for this variable. See below. @item char name[8]; Variable name. The variable name must begin with a capital letter or the at-sign (@samp{@@}). Subsequent characters may also be digits, octothorpes (@samp{#}), dollar signs (@samp{$}), underscores (@samp{_}), or full stops (@samp{.}). The variable name is padded on the right with spaces. The @samp{name} fields should be unique within a system file. System files written by SPSS that contain very long string variables with similar names sometimes contain duplicate names that are later eliminated by resolving the very long string names (@pxref{Very Long String Record}). PSPP handles duplicates by assigning them new, unique names. @item int32 label_len; This field is present only if @code{has_var_label} is set to 1. It is set to the length, in characters, of the variable label. The documented maximum length varies from 120 to 255 based on SPSS version, but some files have been seen with longer labels. PSPP accepts labels of any length. @item char label[]; This field is present only if @code{has_var_label} is set to 1. It has length @code{label_len}, rounded up to the nearest multiple of 32 bits. The first @code{label_len} characters are the variable's variable label. @item flt64 missing_values[]; This field is present only if @code{n_missing_values} is nonzero. It has the same number of 8-byte elements as the absolute value of @code{n_missing_values}. Each element is interpreted as a number for numeric variables (with HIGHEST and LOWEST indicated as described in the chapter introduction). For string variables of width less than 8 bytes, elements are right-padded with spaces; for string variables wider than 8 bytes, only the first 8 bytes of each missing value are specified, with the remainder implicitly all spaces. For discrete missing values, each element represents one missing value. When a range is present, the first element denotes the minimum value in the range, and the second element denotes the maximum value in the range. When a range plus a value are present, the third element denotes the additional discrete missing value. @end table @anchor{System File Output Formats} The @code{print} and @code{write} members of sysfile_variable are output formats coded into @code{int32} types. The least-significant byte of the @code{int32} represents the number of decimal places, and the next two bytes in order of increasing significance represent field width and format type, respectively. The most-significant byte is not used and should be set to zero. Format types are defined as follows: @quotation @multitable {Value} {@code{DATETIME}} @headitem Value @tab Meaning @item 0 @tab Not used. @item 1 @tab @code{A} @item 2 @tab @code{AHEX} @item 3 @tab @code{COMMA} @item 4 @tab @code{DOLLAR} @item 5 @tab @code{F} @item 6 @tab @code{IB} @item 7 @tab @code{PIBHEX} @item 8 @tab @code{P} @item 9 @tab @code{PIB} @item 10 @tab @code{PK} @item 11 @tab @code{RB} @item 12 @tab @code{RBHEX} @item 13 @tab Not used. @item 14 @tab Not used. @item 15 @tab @code{Z} @item 16 @tab @code{N} @item 17 @tab @code{E} @item 18 @tab Not used. @item 19 @tab Not used. @item 20 @tab @code{DATE} @item 21 @tab @code{TIME} @item 22 @tab @code{DATETIME} @item 23 @tab @code{ADATE} @item 24 @tab @code{JDATE} @item 25 @tab @code{DTIME} @item 26 @tab @code{WKDAY} @item 27 @tab @code{MONTH} @item 28 @tab @code{MOYR} @item 29 @tab @code{QYR} @item 30 @tab @code{WKYR} @item 31 @tab @code{PCT} @item 32 @tab @code{DOT} @item 33 @tab @code{CCA} @item 34 @tab @code{CCB} @item 35 @tab @code{CCC} @item 36 @tab @code{CCD} @item 37 @tab @code{CCE} @item 38 @tab @code{EDATE} @item 39 @tab @code{SDATE} @item 40 @tab @code{MTIME} @item 41 @tab @code{YMDHMS} @end multitable @end quotation A few system files have been observed in the wild with invalid @code{write} fields, in particular with value 0. Readers should probably treat invalid @code{print} or @code{write} fields as some default format. @node Value Labels Records @section Value Labels Records The value label records documented in this section are used for numeric and short string variables only. Long string variables may have value labels, but their value labels are recorded using a different record type (@pxref{Long String Value Labels Record}). ReadStat (@pxref{File Header Record}) writes value labels that label a single value more than once. In more detail, it emits value labels whose values are longer than string variables' widths, that are identical in the actual width of the variable, e.g.@: labels for values @code{ABC123} and @code{ABC456} for a string variable with width 3. For files written by this software, PSPP ignores such labels. The value label record has the following format: @example int32 rec_type; int32 label_count; /* @r{Repeated @code{label_cnt} times}. */ char value[8]; char label_len; char label[]; @end example @table @code @item int32 rec_type; Record type. Always set to 3. @item int32 label_count; Number of value labels present in this record. @end table The remaining fields are repeated @code{count} times. Each repetition specifies one value label. @table @code @item char value[8]; A numeric value or a short string value padded as necessary to 8 bytes in length. Its type and width cannot be determined until the following value label variables record (see below) is read. @item char label_len; The label's length, in bytes. The documented maximum length varies from 60 to 120 based on SPSS version. PSPP supports value labels up to 255 bytes long. @item char label[]; @code{label_len} bytes of the actual label, followed by up to 7 bytes of padding to bring @code{label} and @code{label_len} together to a multiple of 8 bytes in length. @end table The value label record is always immediately followed by a value label variables record with the following format: @example int32 rec_type; int32 var_count; int32 vars[]; @end example @table @code @item int32 rec_type; Record type. Always set to 4. @item int32 var_count; Number of variables that the associated value labels from the value label record are to be applied. @item int32 vars[]; A list of 1-based dictionary indexes of variables to which to apply the value labels (@pxref{Dictionary Index}). There are @code{var_count} elements. String variables wider than 8 bytes may not be specified in this list. @end table @node Document Record @section Document Record The document record, if present, has the following format: @example int32 rec_type; int32 n_lines; char lines[][80]; @end example @table @code @item int32 rec_type; Record type. Always set to 6. @item int32 n_lines; Number of lines of documents present. This should be greater than zero, but ReadStats writes system files with zero @code{n_lines}. @item char lines[][80]; Document lines. The number of elements is defined by @code{n_lines}. Lines shorter than 80 characters are padded on the right with spaces. @end table @node Machine Integer Info Record @section Machine Integer Info Record The integer info record, if present, has the following format: @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Data.} */ int32 version_major; int32 version_minor; int32 version_revision; int32 machine_code; int32 floating_point_rep; int32 compression_code; int32 endianness; int32 character_code; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 3. @item int32 size; Size of each piece of data in the data part, in bytes. Always set to 4. @item int32 count; Number of pieces of data in the data part. Always set to 8. @item int32 version_major; PSPP major version number. In version @var{x}.@var{y}.@var{z}, this is @var{x}. @item int32 version_minor; PSPP minor version number. In version @var{x}.@var{y}.@var{z}, this is @var{y}. @item int32 version_revision; PSPP version revision number. In version @var{x}.@var{y}.@var{z}, this is @var{z}. @item int32 machine_code; Machine code. PSPP always set this field to value to -1, but other values may appear. @item int32 floating_point_rep; Floating point representation code. For IEEE 754 systems this is 1. IBM 370 sets this to 2, and DEC VAX E to 3. @item int32 compression_code; Compression code. Always set to 1, regardless of whether or how the file is compressed. @item int32 endianness; Machine endianness. 1 indicates big-endian, 2 indicates little-endian. @item int32 character_code; @anchor{character-code} Character code. The following values have been actually observed in system files: @table @asis @item 1 EBCDIC. @item 2 7-bit ASCII. @item 1250 The @code{windows-1250} code page for Central European and Eastern European languages. @item 1252 The @code{windows-1252} code page for Western European languages. @item 28591 ISO 8859-1. @item 65001 UTF-8. @end table The following additional values are known to be defined: @table @asis @item 3 8-bit ``ASCII''. @item 4 DEC Kanji. @end table Other Windows code page numbers are known to be generally valid. Old versions of SPSS for Unix and Windows always wrote value 2 in this field, regardless of the encoding in use. Newer versions also write the character encoding as a string (see @ref{Character Encoding Record}). @end table @node Machine Floating-Point Info Record @section Machine Floating-Point Info Record The floating-point info record, if present, has the following format: @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Data.} */ flt64 sysmis; flt64 highest; flt64 lowest; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 4. @item int32 size; Size of each piece of data in the data part, in bytes. Always set to 8. @item int32 count; Number of pieces of data in the data part. Always set to 3. @item flt64 sysmis; @itemx flt64 highest; @itemx flt64 lowest; The system missing value, the value used for HIGHEST in missing values, and the value used for LOWEST in missing values, respectively. @xref{System File Format}, for more information. The SPSSWriter library in PHP, which identifies itself as @code{FOM SPSS 1.0.0} in the file header record @code{prod_name} field, writes unexpected values to these fields, but it uses the same values consistently throughout the rest of the file. @end table @node Multiple Response Sets Records @section Multiple Response Sets Records The system file format has two different types of records that represent multiple response sets (@pxref{MRSETS,,,pspp, PSPP Users Guide}). The first type of record describes multiple response sets that can be understood by SPSS before version 14. The second type of record, with a closely related format, is used for multiple dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in version 14. @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char mrsets[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Set to 7 for records that describe multiple response sets understood by SPSS before version 14, or to 19 for records that describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in version 14. @item int32 size; The size of each element in the @code{mrsets} member. Always set to 1. @item int32 count; The total number of bytes in @code{mrsets}. @item char mrsets[]; Zero or more line feeds (byte 0x0a), followed by a series of multiple response sets, each of which consists of the following: @itemize @bullet @item The set's name (an identifier that begins with @samp{$}), in mixed upper and lower case. @item An equals sign (@samp{=}). @item @samp{C} for a multiple category set, @samp{D} for a multiple dichotomy set with CATEGORYLABELS=VARLABELS, or @samp{E} for a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES. @item For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a space, followed by a number expressed as decimal digits, followed by a space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the number is 11; otherwise it is 1.@footnote{This part of the format may not be fully understood, because only a single example of each possibility has been examined.} @item For either kind of multiple dichotomy set, the counted value, as a positive integer count specified as decimal digits, followed by a space, followed by as many string bytes as specified in the count. If the set contains numeric variables, the string consists of the counted integer value expressed as decimal digits. If the set contains string variables, the string contains the counted string value. Either way, the string may be padded on the right with spaces (older versions of SPSS seem to always pad to a width of 8 bytes; newer versions don't). @item A space. @item The multiple response set's label, using the same format as for the counted value for multiple dichotomy sets. A string of length 0 means that the set does not have a label. A string of length 0 is also written if LABELSOURCE=VARLABEL was specified. @item A space. @item The short names of the variables in the set, converted to lowercase, each separated from the previous by a single space. Even though a multiple response set must have at least two variables, some system files contain multiple response sets with no variables or one variable. The source and meaning of these multiple response sets is unknown. (Perhaps they arise from creating a multiple response set then deleting all the variables that it contains?) @item One line feed (byte 0x0a). Sometimes multiple, even hundreds, of line feeds are present. @end itemize @end table Example: Given appropriate variable definitions, consider the following MRSETS command: @example MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55 /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes' /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES VARIABLES=k l m VALUE=34 /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL VARIABLES=n o p VALUE='choice'. @end example The above would generate the following multiple response set record of subtype 7: @example $a=C 10 my mcgroup a b c $b=D2 55 0 g e f d $c=D3 Yes 10 mdgroup #2 h i j @end example It would also generate the following multiple response set record with subtype 19: @example $d=E 1 2 34 13 third mdgroup k l m $e=E 11 6 choice 0 n o p @end example @node Extra Product Info Record @section Extra Product Info Record This optional record appears to contain a text string that describes the program that wrote the file and the source of the data. (This is redundant with the file label and product info found in the file header record.) @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char info[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 10. @item int32 size; The size of each element in the @code{info} member. Always set to 1. @item int32 count; The total number of bytes in @code{info}. @item char info[]; A text string. A product that identifies itself as @code{VOXCO INTERVIEWER 4.3} uses CR-only line ends in this field, rather than the more usual LF-only or CR LF line ends. @end table @node Variable Display Parameter Record @section Variable Display Parameter Record The variable display parameter record, if present, has the following format: @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Repeated @code{count} times}. */ int32 measure; int32 width; /* @r{Not always present.} */ int32 alignment; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 11. @item int32 size; The size of @code{int32}. Always set to 4. @item int32 count; The number of sets of variable display parameters (ordinarily the number of variables in the dictionary), times 2 or 3. @end table The remaining members are repeated @code{count} times, in the same order as the variable records. No element corresponds to variable records that continue long string variables. The meanings of these members are as follows: @table @code @item int32 measure; The measurement type of the variable: @table @asis @item 1 Nominal Scale @item 2 Ordinal Scale @item 3 Continuous Scale @end table SPSS sometimes writes a @code{measure} of 0. PSPP interprets this as nominal scale. @item int32 width; The width of the display column for the variable in characters. This field is present if @var{count} is 3 times the number of variables in the dictionary. It is omitted if @var{count} is 2 times the number of variables. @item int32 alignment; The alignment of the variable for display purposes: @table @asis @item 0 Left aligned @item 1 Right aligned @item 2 Centre aligned @end table @end table @node Long Variable Names Record @section Long Variable Names Record If present, the long variable names record has the following format: @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char var_name_pairs[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 13. @item int32 size; The size of each element in the @code{var_name_pairs} member. Always set to 1. @item int32 count; The total number of bytes in @code{var_name_pairs}. @item char var_name_pairs[]; A list of @var{key}--@var{value} tuples, where @var{key} is the name of a variable, and @var{value} is its long variable name. The @var{key} field is at most 8 bytes long and must match the name of a variable which appears in the variable record (@pxref{Variable Record}). The @var{value} field is at most 64 bytes long. The @var{key} and @var{value} fields are separated by a @samp{=} byte. Each tuple is separated by a byte whose value is 09. There is no trailing separator following the last tuple. The total length is @code{count} bytes. @end table @node Very Long String Record @section Very Long String Record Old versions of SPSS limited string variables to a width of 255 bytes. For backward compatibility with these older versions, the system file format represents a string longer than 255 bytes, called a @dfn{very long string}, as a collection of strings no longer than 255 bytes each. The strings concatenated to make a very long string are called its @dfn{segments}; for consistency, variables other than very long strings are considered to have a single segment. A very long string with a width of @var{w} has @var{n} = (@var{w} + 251) / 252 segments, that is, one segment for every 252 bytes of width, rounding up. It would be logical, then, for each of the segments except the last to have a width of 252 and the last segment to have the remainder, but this is not the case. In fact, each segment except the last has a width of 255 bytes. The last segment has width @var{w} - (@var{n} - 1) * 252; some versions of SPSS make it slightly wider, but not wide enough to make the last segment require another 8 bytes of data. Data is packed tightly into segments of a very long string, 255 bytes per segment. Because 255 bytes of segment data are allocated for every 252 bytes of the very long string's width (approximately), some unused space is left over at the end of the allocated segments. Data in unused space is ignored. Example: Consider a very long string of width 20,000. Such a very long string has 20,000 / 252 = 80 (rounding up) segments. The first 79 segments have width 255; the last segment has width 20,000 - 79 * 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8). The very long string's data is actually stored in the 19,890 bytes in the first 78 segments, plus the first 110 bytes of the 79th segment (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment and all 92 bytes of the 80th segment are unused. The very long string record explains how to stitch together segments to obtain very long string data. For each of the very long string variables in the dictionary, it specifies the name of its first segment's variable and the very long string variable's actual width. The remaining segments immediately follow the named variable in the system file's dictionary. The very long string record, which is present only if the system file contains very long string variables, has the following format: @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char string_lengths[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 14. @item int32 size; The size of each element in the @code{string_lengths} member. Always set to 1. @item int32 count; The total number of bytes in @code{string_lengths}. @item char string_lengths[]; A list of @var{key}--@var{value} tuples, where @var{key} is the name of a variable, and @var{value} is its length. The @var{key} field is at most 8 bytes long and must match the name of a variable which appears in the variable record (@pxref{Variable Record}). The @var{value} field is exactly 5 bytes long. It is a zero-padded, ASCII-encoded string that is the length of the variable. The @var{key} and @var{value} fields are separated by a @samp{=} byte. Tuples are delimited by a two-byte sequence @{00, 09@}. After the last tuple, there may be a single byte 00, or @{00, 09@}. The total length is @code{count} bytes. @end table @node Character Encoding Record @section Character Encoding Record This record, if present, indicates the character encoding for string data, long variable names, variable labels, value labels and other strings in the file. @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char encoding[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 20. @item int32 size; The size of each element in the @code{encoding} member. Always set to 1. @item int32 count; The total number of bytes in @code{encoding}. @item char encoding[]; The name of the character encoding. Normally this will be an official IANA character set name or alias. See @url{http://www.iana.org/assignments/character-sets}. Character set names are not case-sensitive, but SPSS appears to write them in all-uppercase. @end table This record is not present in files generated by older software. See also the @code{character_code} field in the machine integer info record (@pxref{character-code}). When the character encoding record and the machine integer info record are both present, all system files observed in practice indicate the same character encoding, e.g.@: 1252 as @code{character_code} and @code{windows-1252} as @code{encoding}, 65001 and @code{UTF-8}, etc. If, for testing purposes, a file is crafted with different @code{character_code} and @code{encoding}, it seems that @code{character_code} controls the encoding for all strings in the system file before the dictionary termination record, including strings in data (e.g.@: string missing values), and @code{encoding} controls the encoding for strings following the dictionary termination record. @node Long String Value Labels Record @section Long String Value Labels Record This record, if present, specifies value labels for long string variables. @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Repeated up to exactly @code{count} bytes.} */ int32 var_name_len; char var_name[]; int32 var_width; int32 n_labels; long_string_label labels[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 21. @item int32 size; Always set to 1. @item int32 count; The number of bytes following the header until the next header. @item int32 var_name_len; @itemx char var_name[]; The number of bytes in the name of the variable that has long string value labels, plus the variable name itself, which consists of exactly @code{var_name_len} bytes. The variable name is not padded to any particular boundary, nor is it null-terminated. @item int32 var_width; The width of the variable, in bytes, which will be between 9 and 32767. @item int32 n_labels; @itemx long_string_label labels[]; The long string labels themselves. The @code{labels} array contains exactly @code{n_labels} elements, each of which has the following substructure: @example int32 value_len; char value[]; int32 label_len; char label[]; @end example @table @code @item int32 value_len; @itemx char value[]; The string value being labeled. @code{value_len} is the number of bytes in @code{value}; it is equal to @code{var_width}. The @code{value} array is not padded or null-terminated. @item int32 label_len; @itemx char label[]; The label for the string value. @code{label_len}, which must be between 0 and 120, is the number of bytes in @code{label}. The @code{label} array is not padded or null-terminated. @end table @end table @node Long String Missing Values Record @section Long String Missing Values Record This record, if present, specifies missing values for long string variables. @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Repeated up to exactly @code{count} bytes.} */ int32 var_name_len; char var_name[]; char n_missing_values; long_string_missing_value values[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 22. @item int32 size; Always set to 1. @item int32 count; The number of bytes following the header until the next header. @item int32 var_name_len; @itemx char var_name[]; The number of bytes in the name of the long string variable that has missing values, plus the variable name itself, which consists of exactly @code{var_name_len} bytes. The variable name is not padded to any particular boundary, nor is it null-terminated. @item char n_missing_values; The number of missing values, either 1, 2, or 3. (This is, unusually, a single byte instead of a 32-bit number.) @item long_string_missing_value values[]; The missing values themselves. This array contains exactly @code{n_missing_values} elements, each of which has the following substructure: @example int32 value_len; char value[]; @end example @table @code @item int32 value_len; The length of the missing value string, in bytes. This value should be 8, because long string variables are at least 8 bytes wide (by definition), only the first 8 bytes of a long string variable's missing values are allowed to be non-spaces, and any spaces within the first 8 bytes are included in the missing value here. @item char value[]; The missing value string, exactly @code{value_len} bytes, without any padding or null terminator. @end table @end table @node Data File and Variable Attributes Records @section Data File and Variable Attributes Records The data file and variable attributes records represent custom attributes for the system file or for individual variables in the system file, as defined on the DATAFILE ATTRIBUTE (@pxref{DATAFILE ATTRIBUTE,,,pspp, PSPP Users Guide}) and VARIABLE ATTRIBUTE commands (@pxref{VARIABLE ATTRIBUTE,,,pspp, PSPP Users Guide}), respectively. @example /* @r{Header.} */ int32 rec_type; int32 subtype; int32 size; int32 count; /* @r{Exactly @code{count} bytes of data.} */ char attributes[]; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 17 for a data file attribute record or to 18 for a variable attributes record. @item int32 size; The size of each element in the @code{attributes} member. Always set to 1. @item int32 count; The total number of bytes in @code{attributes}. @item char attributes[]; The attributes, in a text-based format. In record subtype 17, this field contains a single attribute set. An attribute set is a sequence of one or more attributes concatenated together. Each attribute consists of a name, which has the same syntax as a variable name, followed by, inside parentheses, a sequence of one or more values. Each value consists of a string enclosed in single quotes (@code{'}) followed by a line feed (byte 0x0a). A value may contain single quote characters, which are not themselves escaped or quoted or required to be present in pairs. There is no apparent way to embed a line feed in a value. There is no distinction between an attribute with a single value and an attribute array with one element. In record subtype 18, this field contains a sequence of one or more variable attribute sets. If more than one variable attribute set is present, each one after the first is delimited from the previous by @code{/}. Each variable attribute set consists of a long variable name, followed by @code{:}, followed by an attribute set with the same syntax as on record subtype 17. System files written by @code{Stata 14.1/-savespss- 1.77 by S.Radyakin} may include multiple records with subtype 18, one per variable that has variable attributes. The total length is @code{count} bytes. @end table @subheading Example A system file produced with the following VARIABLE ATTRIBUTE commands in effect: @example VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34'). VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123'). @end example @noindent will contain a variable attribute record with the following contents: @example 0000 07 00 00 00 12 00 00 00 01 00 00 00 22 00 00 00 |............"...| 0010 64 75 6d 6d 79 3a 66 72 65 64 28 27 32 33 27 0a |dummy:fred('23'.| 0020 27 33 34 27 0a 29 62 65 72 74 28 27 31 32 33 27 |'34'.)bert('123'| 0030 0a 29 |.) | @end example @menu * Variable Roles:: @end menu @node Variable Roles @subsection Variable Roles A variable's role is represented as an attribute named @code{$@@Role}. This attribute has a single element whose values and their meanings are: @table @code @item 0 Input. This, the default, is the most common role. @item 1 Output. @item 2 Both. @item 3 None. @item 4 Partition. @item 5 Split. @end table @node Extended Number of Cases Record @section Extended Number of Cases Record The file header record expresses the number of cases in the system file as an int32 (@pxref{File Header Record}). This record allows the number of cases in the system file to be expressed as a 64-bit number. @example int32 rec_type; int32 subtype; int32 size; int32 count; int64 unknown; int64 ncases64; @end example @table @code @item int32 rec_type; Record type. Always set to 7. @item int32 subtype; Record subtype. Always set to 16. @item int32 size; Size of each element. Always set to 8. @item int32 count; Number of pieces of data in the data part. Alway set to 2. @item int64 unknown; Meaning unknown. Always set to 1. @item int64 ncases64; Number of cases in the file as a 64-bit integer. Presumably this could be -1 to indicate that the number of cases is unknown, for the same reason as @code{ncases} in the file header record, but this has not been observed in the wild. @end table @node Other Informational Records @section Other Informational Records This chapter documents many specific types of extension records are documented here, but others are known to exist. PSPP ignores unknown extension records when reading system files. The following extension record subtypes have also been observed, with the following believed meanings: @table @asis @item 5 A set of grouped variables (according to Aapi H@"am@"al@"ainen). @item 6 Date info, probably related to USE (according to Aapi H@"am@"al@"ainen). @item 12 A UUID in the format described in RFC 4122. Only two examples observed, both written by SPSS 13, and in each case the UUID contained both upper and lower case. @item 24 XML that describes how data in the file should be displayed on-screen. @end table @node Dictionary Termination Record @section Dictionary Termination Record The dictionary termination record separates all other records from the data records. @example int32 rec_type; int32 filler; @end example @table @code @item int32 rec_type; Record type. Always set to 999. @item int32 filler; Ignored padding. Should be set to 0. @end table @node Data Record @section Data Record The data record must follow all other records in the system file. Every system file must have a data record that specifies data for at least one case. The format of the data record varies depending on the value of @code{compression} in the file header record: @table @asis @item 0: no compression Data is arranged as a series of 8-byte elements. Each element corresponds to the variable declared in the respective variable record (@pxref{Variable Record}). Numeric values are given in @code{flt64} format; string values are literal characters string, padded on the right when necessary to fill out 8-byte units. @item 1: bytecode compression The first 8 bytes of the data record is divided into a series of 1-byte command codes. These codes have meanings as described below: @table @asis @item 0 Ignored. If the program writing the system file accumulates compressed data in blocks of fixed length, 0 bytes can be used to pad out extra bytes remaining at the end of a fixed-size block. @item 1 through 251 A number with value @var{code} - @var{bias}, where @var{code} is the value of the compression code and @var{bias} is the variable @code{bias} from the file header. For example, code 105 with bias 100.0 (the normal value) indicates a numeric variable of value 5. A code of 0 (after subtracting the bias) in a string field encodes null bytes. This is unusual, since a string field normally encodes text data, but it exists in real system files. @item 252 End of file. This code may or may not appear at the end of the data stream. PSPP always outputs this code but its use is not required. @item 253 A numeric or string value that is not compressible. The value is stored in the 8 bytes following the current block of command bytes. If this value appears twice in a block of command bytes, then it indicates the second group of 8 bytes following the command bytes, and so on. @item 254 An 8-byte string value that is all spaces. @item 255 The system-missing value. @end table The end of the 8-byte group of bytecodes is followed by any 8-byte blocks of non-compressible values indicated by code 253. After that follows another 8-byte group of bytecodes, then those bytecodes' non-compressible values. The pattern repeats to the end of the file or a code with value 252. @item 2: ZLIB compression The data record consists of the following, in order: @itemize @bullet @item ZLIB data header, 24 bytes long. @item One or more variable-length blocks of ZLIB compressed data. @item ZLIB data trailer, with a 24-byte fixed header plus an additional 24 bytes for each preceding ZLIB compressed data block. @end itemize The ZLIB data header has the following format: @example int64 zheader_ofs; int64 ztrailer_ofs; int64 ztrailer_len; @end example @table @code @item int64 zheader_ofs; The offset, in bytes, of the beginning of this structure within the system file. @item int64 ztrailer_ofs; The offset, in bytes, of the first byte of the ZLIB data trailer. @item int64 ztrailer_len; The number of bytes in the ZLIB data trailer. This and the previous field sum to the size of the system file in bytes. @end table The data header is followed by @code{(ztrailer_len - 24) / 24} ZLIB compressed data blocks. Each ZLIB compressed data block begins with a ZLIB header as specified in RFC@tie{}1950, e.g.@: hex bytes @code{78 01} (the only header yet observed in practice). Each block decompresses to a fixed number of bytes (in practice only @code{0x3ff000}-byte blocks have been observed), except that the last block of data may be shorter. The last ZLIB compressed data block gends just before offset @code{ztrailer_ofs}. The result of ZLIB decompression is bytecode compressed data as described above for compression format 1. The ZLIB data trailer begins with the following 24-byte fixed header: @example int64 bias; int64 zero; int32 block_size; int32 n_blocks; @end example @table @code @item int64 int_bias; The compression bias as a negative integer, e.g.@: if @code{bias} in the file header record is 100.0, then @code{int_bias} is @minus{}100 (this is the only value yet observed in practice). @item int64 zero; Always observed to be zero. @item int32 block_size; The number of bytes in each ZLIB compressed data block, except possibly the last, following decompression. Only @code{0x3ff000} has been observed so far. @item int32 n_blocks; The number of ZLIB compressed data blocks, always exactly @code{(ztrailer_len - 24) / 24}. @end table The fixed header is followed by @code{n_blocks} 24-byte ZLIB data block descriptors, each of which describes the compressed data block corresponding to its offset. Each block descriptor has the following format: @example int64 uncompressed_ofs; int64 compressed_ofs; int32 uncompressed_size; int32 compressed_size; @end example @table @code @item int64 uncompressed_ofs; The offset, in bytes, that this block of data would have in a similar system file that uses compression format 1. This is @code{zheader_ofs} in the first block descriptor, and in each succeeding block descriptor it is the sum of the previous desciptor's @code{uncompressed_ofs} and @code{uncompressed_size}. @item int64 compressed_ofs; The offset, in bytes, of the actual beginning of this compressed data block. This is @code{zheader_ofs + 24} in the first block descriptor, and in each succeeding block descriptor it is the sum of the previous descriptor's @code{compressed_ofs} and @code{compressed_size}. The final block descriptor's @code{compressed_ofs} and @code{compressed_size} sum to @code{ztrailer_ofs}. @item int32 uncompressed_size; The number of bytes in this data block, after decompression. This is @code{block_size} in every data block except the last, which may be smaller. @item int32 compressed_size; The number of bytes in this data block, as stored compressed in this system file. @end table @end table @setfilename ignored