1[/==============================================================================
2    Copyright (C) 2001-2011 Hartmut Kaiser
3    Copyright (C) 2001-2011 Joel de Guzman
4
5    Distributed under the Boost Software License, Version 1.0. (See accompanying
6    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
7===============================================================================/]
8
9[section:lexer_concepts Lexer Concepts]
10
11__lex__ components fall into a couple of generalized __concepts__. The
12/Lexer/ is the most fundamental concept. All __lex__ components are
13models of the /Lexer/ concept. /PrimitiveLexer/, /UnaryLexer/,
14and /NaryLexer/ are all refinements of the /Lexer/ concept.
15
16The following sections provide details on these concepts.
17
18[/////////////////////////////////////////////////////////////////////////////]
19[section Lexer]
20
21[heading Description]
22
23The /Lexer/ is the most fundamental concept. A Lexer has a member
24function, `collect`, that accepts a token definition container `Def`, and a
25the name of the lexer state the token definitions of the lexer component need
26to be added to (a string). It doesn't return anything (return type is `void`).
27Each Lexer can represent a specific pattern or algorithm, or it
28can be a more complex lexer component formed as a composition of other Lexer's.
29Additionally, a Lexer exposes a member `add_actions`, that accepts the token
30definition container `Def`, while returning nothing (again, the returned type
31is `void`).
32
33[variablelist Notation
34    [[`l`]              [A `Lexer`.]]
35    [[`L`]              [A `Lexer` type.]]
36    [[`Def`]            [A token definition container type.]]
37    [[`State`]          [A type used to represent lexer state names.]]
38]
39
40[heading Valid Expressions]
41
42In the expressions below, the behavior of the lexer component, `l`, is left
43unspecified in the base `Lexer` concept. These are specified in subsequent,
44more refined concepts and by the actual models thereof.
45
46For any Lexer the following expressions must be valid:
47
48[table
49    [[Expression]              [Semantics]                        [Return type]]
50    [[`l.collect(def, state, targetstate)`]
51                               [Add all token definitions provided
52                                by this Lexer instance to the lexer
53                                state `state` of the token definition
54                                container `def`. After matching this token, the
55                                lexer should be switched into the state
56                                `targetstate` (optional)]         [`void`]]
57    [[`l.add_actions(def)`]    [Add all semantic actions provided
58                                by this Lexer instance to the token
59                                definition container `def`.]      [`void`]]
60]
61
62[heading Type Expressions]
63
64[table
65    [[Expression]                                   [Description]]
66    [[`traits::is_lexer<L>::type`]                  [Metafunction that evaluates to `mpl::true_` if
67                                                     a certain type, `L` is a Lexer, `mpl::false_`
68                                                     otherwise (See __mpl_boolean_constant__).]]
69]
70
71[heading Postcondition]
72
73Upon return from `l.collect` the following post conditions should hold:
74
75* On return, `def` holds all token definitions defined in the Lexer, `l`. This
76  includes all Lexer's contained inside `l`.
77
78Upon return from `l.add_actions` the following post conditions should hold:
79
80* On return, `def` holds all semantic actions correctly associated with the
81  corresponding token definitions as defined in the Lexer, `l`. This
82  includes all semantic actions defined by the Lexer's contained inside `l`.
83
84[heading Models]
85
86All lexer components in __lex__ are models of the /Lexer/ concept.
87
88[endsect] [/ Lexer Concept]
89
90[/////////////////////////////////////////////////////////////////////////////]
91[section PrimitiveLexer]
92
93[heading Description]
94
95/PrimitiveLexer/ is the most basic building block that the client uses
96to build more complex lexer components.
97
98[heading Refinement of]
99
100[:__lexer_concept__]
101
102[heading Type Expressions]
103
104[table
105    [[Expression]                                   [Description]]
106    [[`traits::is_primitive_lexer<L>::type`]        [Metafunction that evaluates to `mpl::true_` if
107                                                    a certain type, `L`, is a PrimitiveLexer, `mpl::false_`
108                                                    otherwise (See __mpl_boolean_constant__).]]
109]
110
111[heading Models]
112
113The following lexer components conform to this model:
114
115* character literals (i.e. `'x'`), `char_`,
116* string literals (`"abc"`), `std::basic_string<>`, `string`
117
118__fixme__ Add more links to /PrimitiveLexer/ models here.
119
120[endsect] [/ PrimitiveLexer Concept]
121
122[/////////////////////////////////////////////////////////////////////////////]
123[section UnaryLexer]
124
125[heading Description]
126
127/UnaryLexer/ is a composite lexer component that has a single subject. The
128UnaryLexer may change the behavior of its subject following the
129__delegate_pattern__.
130
131[heading Refinement of]
132
133[:__lexer_concept__]
134
135[variablelist Notation
136    [[`l`]              [A UnaryLexer.]]
137    [[`L`]              [A UnaryLexer type.]]
138]
139
140[heading Valid Expressions]
141
142In addition to the requirements defined in __lexer_concept__, for any
143UnaryLexer the following must be met:
144
145[table
146    [[Expression]       [Semantics]                 [Return type]]
147    [[`l.subject`]      [Subject lexer component.]  [__lexer_concept__]]
148]
149
150[heading Type Expressions]
151
152[table
153    [[Expression]           [Description]]
154    [[`L::subject_type`]    [The subject lexer component type.]]
155    [[`traits::is_unary_lexer<L>::type`]       [Metafunction that evaluates to `mpl::true_` if
156                                                a certain type, `L` is a UnaryLexer, `mpl::false_`
157                                                otherwise (See __mpl_boolean_constant__).]]
158]
159
160[heading Invariants]
161
162For any UnaryLexer, `L`, the following invariant always holds:
163
164* `traits::is_lexer<L::subject_type>::type` evaluates to `mpl::true_`
165
166[heading Models]
167
168The following lexer components conform to this model:
169
170* action lexer component (allowing to attach semantic actions)
171
172__fixme__ Add more links to models of UnaryLexer concept
173
174[endsect] [/ UnaryLexer Concept]
175
176[/////////////////////////////////////////////////////////////////////////////]
177[section NaryLexer]
178
179[heading Description]
180
181/NaryLexer/ is a composite lexer component that has one or more subjects. The
182NaryLexer allows its subjects to be treated in the same way as a single
183instance of a __lexer_concept__ following the __composite_pattern__.
184
185[heading Refinement of]
186
187[:__lexer_concept__]
188
189[variablelist Notation
190    [[`l`]              [A NaryLexer.]]
191    [[`L`]              [A NaryLexer type.]]
192]
193
194[heading Valid Expressions]
195
196In addition to the requirements defined in __lexer_concept__, for any
197NaryLexer the following must be met:
198
199[table
200    [[Expression]       [Semantics]                 [Return type]]
201    [[`l.elements`]     [The tuple of elements.]    [A __fusion__ Sequence of __lexer_concept__ types.]]
202]
203
204[heading Type Expressions]
205
206[table
207    [[Expression]           [Description]]
208    [[`l.elements_type`]    [Elements tuple type.]]
209    [[`traits::is_nary_lexer<L>::type`]             [Metafunction that evaluates to `mpl::true_` if
210                                                     a certain type, `L` is a NaryLexer, `mpl::false_`
211                                                     otherwise (See __mpl_boolean_constant__).]]
212]
213
214[heading Invariants]
215
216For each element, `E`, in any NaryLexer, `L`, the following
217invariant always holds:
218
219* `traits::is_lexer<E>::type` evaluates to `mpl::true_`
220
221[heading Models]
222
223The following lexer components conform to this model:
224
225* lexer sequence component
226
227__fixme__ Add more links to models of NaryLexer concept
228
229[endsect] [/ NaryLexer Concept]
230
231[endsect]
232