1With ``-f`` ``--storable-state`` option re2c generates a lexer that can store
2its current state, return to the caller, and later resume operations exactly
3where it left off. The default mode of operation in re2c is a "pull" model,
4in which the lexer "pulls" more input whenever it needs it. This may be
5unacceptable in cases when the input becomes available piece by piece (for
6example, if the lexer is invoked by the parser, or if the lexer program
7communicates via a socket protocol with some other program that must wait for a
8reply from the lexer before it transmits the next message). Storable state
9feature is intended exactly for such cases: it allows one to generate lexers that
10work in a "push" model. When the lexer needs more input, it stores its state and
11returns to the caller. Later, when more input becomes available, the caller
12resumes the lexer exactly where it stopped. There are a few changes necessary
13compared to the "pull" model:
14
15* Define ``YYSETSTATE()`` and ``YYGETSTATE(state)`` promitives.
16
17* Define ``yych``, ``yyaccept`` and ``state`` variables as a part of persistent
18  lexer state. The ``state`` variable should be initialized to ``-1``.
19
20* ``YYFILL`` should return to the outer program instead of trying to supply more
21  input. Return code should indicate that lexer needs more input.
22
23* The outer program should recognize situations when lexer needs more input and
24  respond appropriately.
25
26* Use ``/*!getstate:re2c*/`` directive if it is necessary to execute any code
27  before entering the lexer.
28
29* Use configurations ``state:abort`` and ``state:nextlabel`` to further tweak
30  the generated code.
31
32Here is an example of a "push"-model lexer that reads input from ``stdin`` and
33expects a sequence of words separated by spaces and newlines. The lexer loops
34forever, waiting for more input. It can be terminated by sending a special EOF
35token --- a word "stop", in which case the lexer terminates successfully and
36prints the number of words it has seen. Abnormal termination happens in case of
37a syntax error, premature end of input (without the "stop" word) or in case the
38buffer is too small to hold a lexeme (for example, if one of the words exceeds
39buffer size). Premature end of input happens in case the lexer fails to read any
40input while being in the initial state --- this is the only case when EOF rule
41matches. Note that the lexer may call ``YYFILL`` twice before terminating (and
42thus require hitting ``Ctrl+D`` a few times). First time ``YYFILL`` is called
43when the lexer expects continuation of the current greedy lexeme (either a word
44or a whitespace sequence). If ``YYFILL`` fails, the lexer knows that it has
45reached the end of the current lexeme and executes the corresponding semantic
46action. The action jumps to the beginning of the loop, the lexer enters the
47initial state and calls ``YYFILL`` once more. If it fails, the lexer matches EOF
48rule. (Alternatively EOF rule can be used for termination instead of a special
49EOF lexeme.)
50