Welcome to the OpenBIOS tokenizer Toke.

This README provides you with a short description of the tokenizer and
of how to use it.

---------------------------------------------------------------------

Table of contents:

1. What is the OpenBIOS tokenizer?
2. What is required to build the OpenBIOS tokenizer?
3. How to use the OpenBIOS tokenizer
4. Toke's Special Features not described in IEEE 1275-1994
5. Text Strings
6. Contact

---------------------------------------------------------------------

1. What is the OpenBIOS tokenizer?

   toke is a GPLed FCode tokenizer. It can tokenize (assemble) fcode
   source to bytecode files as described by the IEEE 1275-1994 standard.

   This program is compliant to IEEE 1275-1994.

   Bytecode files normally contain Open Firmware drivers or other
   packages for use with an Open Firmware compliant system.


2. What is required to build the OpenBIOS tokenizer?

   toke should build with any ANSI compliant C compiler.

   For building toke on certain platforms you might have to adjust the
   Makefile. If so, please contact Stefan Reinauer <stepan@openbios.org>

   Build toke by just enter "make". To clean up an existing build,
   use "make clean" or "make distclean"


3. How to use the OpenBIOS tokenizer

   tokenize an fcode source file with
	toke [-v|--verbose] [-i|--ignore-errors] <file.4th> [<file2.4th>]

   Get help with:
	toke [-h|--help]

   The file extension will be changed to .fc. If no file extension
   exists, .fc will be appended. If -i is specified, toke continues
   on errors, producing invalid fcode binaries.


4. Toke's Special Features not described in IEEE 1275-1994

   In addition to the standard Open Firmware words, some additional
   commands have been added to make life easier and for compatibility
   to other, mostly commercial, fcode tokenizers. These are:

4.1 NEXT-FCODE

   In tokenizer[ .. ]tokenizer context there is an additional control
   word besides emit-byte: next-fcode. It sets the increasing fcode
   number used to emit new fcode words to the specified value. Thus
   allowing sparse fcode numbering and overwriting previously defined
   fcode words.

   This could look like:
   -------------------------------------------
       \ next emitted fcode is 0x053, 2dup.
       tokenizer[ 053 next-fcode ]tokenizer
       \ now define the word.
       : 2dup over over ;
   -------------------------------------------

   Note: Toke is capable of creating fcode files that implement words
   defined by the IEEE 1275-1994 standard itself. This is not necessarily
   supported by every Open Firmware implementation.


4.2 FCODE-VERSION2

   This word generates an FCode header using START1. It should be used
   with FCODE-END.


4.3 FCODE-END

   This word creates an END0 FCode and fixes up the FCode header
   information. Use this with FCODE-VERSION2


4.4 PCI-HEADER

   This word creates a PCI option ROM header. PCI-HEADER expects 3
   values on the top of the stack when it is invoked: Vendor#, Device#
   and Class-Code.

   These values are used for the according fields in the PCI Data
   structure that is emitted by PCI-HEADER.  To include these values
   put them on the stack with TOKENIZER[ and ]TOKENIZER.


4.5 PCI-END / PCI-HEADER-END

   This word completes the PCI header created by PCI-HEADER. It fills out
   the image to a multiple of 512 bytes, and sets the missing values in
   the PCI header (image-length, revision).

   Example:
   HEX
   TOKENIZER[ 1234 5678 ABCDEF ]TOKENIZER
   PCI-HEADER \ generate PCI option rom header
	FCODE-VERSION2 \ generate FCode header (within PCI image)
        ...
        ...
	FCODE-END \ terminate FCode in image
   PCI-END \ complete the PCI header and image.


4.6 PCI-REVISION / SET-REV-LEVEL

   Use this word to change the revision field of the PCI header. Like
   PCI-HEADER, PCI-REVISION takes it's value from the stack. You can
   write: TOKENIZER[ 23 ]TOKENIZER PCI-REVISION


4.7 NOT-LAST-IMAGE

   Normally Toke assumes that a PCI image generated by PCI-HEADER is
   the only ROM image in the (EEP)ROM it will be written to. With
   NOT-LAST-IMAGE Toke sets a flag in the PCI header that other images
   will follow in the same ROM.


4.8 FCODE-DATE

   FCODE-DATE creates an FCode string that contains the date of the
   tokenization. This string could be used to create a device tree
   property that can be inspected by  drivers, etc. to check when the
   image was created. The format of the string emitted by FCODE-DATE
   is MM/DD.YYYY


4.9 FCODE-TIME

   FCODE-TIME creates an FCode string that contains the time of the
   tokenization.  This string could be used to create a device tree
   property that can be inspected by drivers, etc. to check when the
   image was created. The format of the string emitted by FCODE-TIME
   is HH:MM:SS


4.10 ENCODE-FILE

   encode a binary file. Use as: encode-file filename


5. Text Strings

   From "Writing FCode":
   Escape Sequences in Text Strings

   -----------------------------------------------------------
   Syntax          Function
   -----------------------------------------------------------
   ""              quote (")
   "n              newline
   "r              carriage return
   "t              tab
   "f              formfeed
   "l              linefeed
   "b              backspace
   "!              bell
   "^x             control x, where x is any printable
                   character
   "(hh hh)        Sequence of bytes, one byte for each pair
                   of hex digits hh . Non-hex characters will
                   be ignored

   " followed by any other printable character not mentioned
   above is equivalent to that character.

   "( means to start parsing pairs of hexadecimal digits as
   one or more 8-bit characters in the range 0x00 through
   0xFF, delimited by a trailing ) and ignoring
   non-hexadecimal digits between pairs of hexadecimal
   digits. Both uppercase and lowercase hexadecimal digits are
   recognized. Since non-hex characters (such as space or
   comma) are ignored between "( and ), these characters make
   useful delimiters. (The "makearray" tool can be used in
   conjunction with this syntax to easily incorporate large
   binary data fields into any FCode Program.)

   Any characters thus recognized are appended to any previous
   text in the string being assembled. After the ) is
   recognized, text assembly continues until a trailing
   "<whitespace>.

   For example:

   " This is "(01 32,8e)abc"nA test! "!"! abcd""hijk"^bl"
               ^^^^^^^^    ^         ^ ^      ^      ^
                3 bytes    newline   2 bells  "      control b

6. Contact

   Ideas, bug reports, patches, contributions are welcome.

        Stefan Reinauer <stepan@openbios.org>