1\input texinfo.tex @c -*-texinfo-*- 2@c $NetBSD: flex.texi,v 1.1.1.2 2013/04/06 14:05:50 christos Exp $ 3@c %**start of header 4@setfilename flex.info 5@include version.texi 6@settitle Lexical Analysis With Flex, for Flex @value{VERSION} 7@set authors Vern Paxson, Will Estes and John Millaway 8@c "Macro Hooks" index 9@defindex hk 10@c "Options" index 11@defindex op 12@dircategory Programming 13@direntry 14* flex: (flex). Fast lexical analyzer generator (lex replacement). 15@end direntry 16@c %**end of header 17 18@copying 19 20The flex manual is placed under the same licensing conditions as the 21rest of flex: 22 23Copyright @copyright{} 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012 24The Flex Project. 25 26Copyright @copyright{} 1990, 1997 The Regents of the University of California. 27All rights reserved. 28 29This code is derived from software contributed to Berkeley by 30Vern Paxson. 31 32The United States Government has rights in this work pursuant 33to contract no. DE-AC03-76SF00098 between the United States 34Department of Energy and the University of California. 35 36Redistribution and use in source and binary forms, with or without 37modification, are permitted provided that the following conditions 38are met: 39 40@enumerate 41@item 42 Redistributions of source code must retain the above copyright 43notice, this list of conditions and the following disclaimer. 44 45@item 46Redistributions in binary form must reproduce the above copyright 47notice, this list of conditions and the following disclaimer in the 48documentation and/or other materials provided with the distribution. 49@end enumerate 50 51Neither the name of the University nor the names of its contributors 52may be used to endorse or promote products derived from this software 53without specific prior written permission. 54 55THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR 56IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED 57WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 58PURPOSE. 59@end copying 60 61@titlepage 62@title Lexical Analysis with Flex 63@subtitle Edition @value{EDITION}, @value{UPDATED} 64@author @value{authors} 65@page 66@vskip 0pt plus 1filll 67@insertcopying 68@end titlepage 69@contents 70@ifnottex 71@node Top, Copyright, (dir), (dir) 72@top flex 73 74This manual describes @code{flex}, a tool for generating programs that 75perform pattern-matching on text. The manual includes both tutorial and 76reference sections. 77 78This edition of @cite{The flex Manual} documents @code{flex} version 79@value{VERSION}. It was last updated on @value{UPDATED}. 80 81This manual was written by @value{authors}. 82 83@menu 84* Copyright:: 85* Reporting Bugs:: 86* Introduction:: 87* Simple Examples:: 88* Format:: 89* Patterns:: 90* Matching:: 91* Actions:: 92* Generated Scanner:: 93* Start Conditions:: 94* Multiple Input Buffers:: 95* EOF:: 96* Misc Macros:: 97* User Values:: 98* Yacc:: 99* Scanner Options:: 100* Performance:: 101* Cxx:: 102* Reentrant:: 103* Lex and Posix:: 104* Memory Management:: 105* Serialized Tables:: 106* Diagnostics:: 107* Limitations:: 108* Bibliography:: 109* FAQ:: 110* Appendices:: 111* Indices:: 112 113@detailmenu 114 --- The Detailed Node Listing --- 115 116Format of the Input File 117 118* Definitions Section:: 119* Rules Section:: 120* User Code Section:: 121* Comments in the Input:: 122 123Scanner Options 124 125* Options for Specifying Filenames:: 126* Options Affecting Scanner Behavior:: 127* Code-Level And API Options:: 128* Options for Scanner Speed and Size:: 129* Debugging Options:: 130* Miscellaneous Options:: 131 132Reentrant C Scanners 133 134* Reentrant Uses:: 135* Reentrant Overview:: 136* Reentrant Example:: 137* Reentrant Detail:: 138* Reentrant Functions:: 139 140The Reentrant API in Detail 141 142* Specify Reentrant:: 143* Extra Reentrant Argument:: 144* Global Replacement:: 145* Init and Destroy Functions:: 146* Accessor Methods:: 147* Extra Data:: 148* About yyscan_t:: 149 150Memory Management 151 152* The Default Memory Management:: 153* Overriding The Default Memory Management:: 154* A Note About yytext And Memory:: 155 156Serialized Tables 157 158* Creating Serialized Tables:: 159* Loading and Unloading Serialized Tables:: 160* Tables File Format:: 161 162FAQ 163 164* When was flex born?:: 165* How do I expand backslash-escape sequences in C-style quoted strings?:: 166* Why do flex scanners call fileno if it is not ANSI compatible?:: 167* Does flex support recursive pattern definitions?:: 168* How do I skip huge chunks of input (tens of megabytes) while using flex?:: 169* Flex is not matching my patterns in the same order that I defined them.:: 170* My actions are executing out of order or sometimes not at all.:: 171* How can I have multiple input sources feed into the same scanner at the same time?:: 172* Can I build nested parsers that work with the same input file?:: 173* How can I match text only at the end of a file?:: 174* How can I make REJECT cascade across start condition boundaries?:: 175* Why cant I use fast or full tables with interactive mode?:: 176* How much faster is -F or -f than -C?:: 177* If I have a simple grammar cant I just parse it with flex?:: 178* Why doesn't yyrestart() set the start state back to INITIAL?:: 179* How can I match C-style comments?:: 180* The period isn't working the way I expected.:: 181* Can I get the flex manual in another format?:: 182* Does there exist a "faster" NDFA->DFA algorithm?:: 183* How does flex compile the DFA so quickly?:: 184* How can I use more than 8192 rules?:: 185* How do I abandon a file in the middle of a scan and switch to a new file?:: 186* How do I execute code only during initialization (only before the first scan)?:: 187* How do I execute code at termination?:: 188* Where else can I find help?:: 189* Can I include comments in the "rules" section of the file?:: 190* I get an error about undefined yywrap().:: 191* How can I change the matching pattern at run time?:: 192* How can I expand macros in the input?:: 193* How can I build a two-pass scanner?:: 194* How do I match any string not matched in the preceding rules?:: 195* I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: 196* Is there a way to make flex treat NULL like a regular character?:: 197* Whenever flex can not match the input it says "flex scanner jammed".:: 198* Why doesn't flex have non-greedy operators like perl does?:: 199* Memory leak - 16386 bytes allocated by malloc.:: 200* How do I track the byte offset for lseek()?:: 201* How do I use my own I/O classes in a C++ scanner?:: 202* How do I skip as many chars as possible?:: 203* deleteme00:: 204* Are certain equivalent patterns faster than others?:: 205* Is backing up a big deal?:: 206* Can I fake multi-byte character support?:: 207* deleteme01:: 208* Can you discuss some flex internals?:: 209* unput() messes up yy_at_bol:: 210* The | operator is not doing what I want:: 211* Why can't flex understand this variable trailing context pattern?:: 212* The ^ operator isn't working:: 213* Trailing context is getting confused with trailing optional patterns:: 214* Is flex GNU or not?:: 215* ERASEME53:: 216* I need to scan if-then-else blocks and while loops:: 217* ERASEME55:: 218* ERASEME56:: 219* ERASEME57:: 220* Is there a repository for flex scanners?:: 221* How can I conditionally compile or preprocess my flex input file?:: 222* Where can I find grammars for lex and yacc?:: 223* I get an end-of-buffer message for each character scanned.:: 224* unnamed-faq-62:: 225* unnamed-faq-63:: 226* unnamed-faq-64:: 227* unnamed-faq-65:: 228* unnamed-faq-66:: 229* unnamed-faq-67:: 230* unnamed-faq-68:: 231* unnamed-faq-69:: 232* unnamed-faq-70:: 233* unnamed-faq-71:: 234* unnamed-faq-72:: 235* unnamed-faq-73:: 236* unnamed-faq-74:: 237* unnamed-faq-75:: 238* unnamed-faq-76:: 239* unnamed-faq-77:: 240* unnamed-faq-78:: 241* unnamed-faq-79:: 242* unnamed-faq-80:: 243* unnamed-faq-81:: 244* unnamed-faq-82:: 245* unnamed-faq-83:: 246* unnamed-faq-84:: 247* unnamed-faq-85:: 248* unnamed-faq-86:: 249* unnamed-faq-87:: 250* unnamed-faq-88:: 251* unnamed-faq-90:: 252* unnamed-faq-91:: 253* unnamed-faq-92:: 254* unnamed-faq-93:: 255* unnamed-faq-94:: 256* unnamed-faq-95:: 257* unnamed-faq-96:: 258* unnamed-faq-97:: 259* unnamed-faq-98:: 260* unnamed-faq-99:: 261* unnamed-faq-100:: 262* unnamed-faq-101:: 263* What is the difference between YYLEX_PARAM and YY_DECL?:: 264* Why do I get "conflicting types for yylex" error?:: 265* How do I access the values set in a Flex action from within a Bison action?:: 266 267Appendices 268 269* Makefiles and Flex:: 270* Bison Bridge:: 271* M4 Dependency:: 272* Common Patterns:: 273 274Indices 275 276* Concept Index:: 277* Index of Functions and Macros:: 278* Index of Variables:: 279* Index of Data Types:: 280* Index of Hooks:: 281* Index of Scanner Options:: 282 283@end detailmenu 284@end menu 285@end ifnottex 286@node Copyright, Reporting Bugs, Top, Top 287@chapter Copyright 288 289@cindex copyright of flex 290@cindex distributing flex 291@insertcopying 292 293@node Reporting Bugs, Introduction, Copyright, Top 294@chapter Reporting Bugs 295 296@cindex bugs, reporting 297@cindex reporting bugs 298 299If you find a bug in @code{flex}, please report it using 300the SourceForge Bug Tracking facilities which can be found on 301@url{http://sourceforge.net/projects/flex,flex's SourceForge Page}. 302 303@node Introduction, Simple Examples, Reporting Bugs, Top 304@chapter Introduction 305 306@cindex scanner, definition of 307@code{flex} is a tool for generating @dfn{scanners}. A scanner is a 308program which recognizes lexical patterns in text. The @code{flex} 309program reads the given input files, or its standard input if no file 310names are given, for a description of a scanner to generate. The 311description is in the form of pairs of regular expressions and C code, 312called @dfn{rules}. @code{flex} generates as output a C source file, 313@file{lex.yy.c} by default, which defines a routine @code{yylex()}. 314This file can be compiled and linked with the flex runtime library to 315produce an executable. When the executable is run, it analyzes its 316input for occurrences of the regular expressions. Whenever it finds 317one, it executes the corresponding C code. 318 319@node Simple Examples, Format, Introduction, Top 320@chapter Some Simple Examples 321 322First some simple examples to get the flavor of how one uses 323@code{flex}. 324 325@cindex username expansion 326The following @code{flex} input specifies a scanner which, when it 327encounters the string @samp{username} will replace it with the user's 328login name: 329 330@example 331@verbatim 332 %% 333 username printf( "%s", getlogin() ); 334@end verbatim 335@end example 336 337@cindex default rule 338@cindex rules, default 339By default, any text not matched by a @code{flex} scanner is copied to 340the output, so the net effect of this scanner is to copy its input file 341to its output with each occurrence of @samp{username} expanded. In this 342input, there is just one rule. @samp{username} is the @dfn{pattern} and 343the @samp{printf} is the @dfn{action}. The @samp{%%} symbol marks the 344beginning of the rules. 345 346Here's another simple example: 347 348@cindex counting characters and lines 349@example 350@verbatim 351 int num_lines = 0, num_chars = 0; 352 353 %% 354 \n ++num_lines; ++num_chars; 355 . ++num_chars; 356 357 %% 358 359 int main() 360 { 361 yylex(); 362 printf( "# of lines = %d, # of chars = %d\n", 363 num_lines, num_chars ); 364 } 365@end verbatim 366@end example 367 368This scanner counts the number of characters and the number of lines in 369its input. It produces no output other than the final report on the 370character and line counts. The first line declares two globals, 371@code{num_lines} and @code{num_chars}, which are accessible both inside 372@code{yylex()} and in the @code{main()} routine declared after the 373second @samp{%%}. There are two rules, one which matches a newline 374(@samp{\n}) and increments both the line count and the character count, 375and one which matches any character other than a newline (indicated by 376the @samp{.} regular expression). 377 378A somewhat more complicated example: 379 380@cindex Pascal-like language 381@example 382@verbatim 383 /* scanner for a toy Pascal-like language */ 384 385 %{ 386 /* need this for the call to atof() below */ 387 #include <math.h> 388 %} 389 390 DIGIT [0-9] 391 ID [a-z][a-z0-9]* 392 393 %% 394 395 {DIGIT}+ { 396 printf( "An integer: %s (%d)\n", yytext, 397 atoi( yytext ) ); 398 } 399 400 {DIGIT}+"."{DIGIT}* { 401 printf( "A float: %s (%g)\n", yytext, 402 atof( yytext ) ); 403 } 404 405 if|then|begin|end|procedure|function { 406 printf( "A keyword: %s\n", yytext ); 407 } 408 409 {ID} printf( "An identifier: %s\n", yytext ); 410 411 "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext ); 412 413 "{"[\^{}}\n]*"}" /* eat up one-line comments */ 414 415 [ \t\n]+ /* eat up whitespace */ 416 417 . printf( "Unrecognized character: %s\n", yytext ); 418 419 %% 420 421 int main( int argc, char **argv ) 422 { 423 ++argv, --argc; /* skip over program name */ 424 if ( argc > 0 ) 425 yyin = fopen( argv[0], "r" ); 426 else 427 yyin = stdin; 428 429 yylex(); 430 } 431@end verbatim 432@end example 433 434This is the beginnings of a simple scanner for a language like Pascal. 435It identifies different types of @dfn{tokens} and reports on what it has 436seen. 437 438The details of this example will be explained in the following 439sections. 440 441@node Format, Patterns, Simple Examples, Top 442@chapter Format of the Input File 443 444 445@cindex format of flex input 446@cindex input, format of 447@cindex file format 448@cindex sections of flex input 449 450The @code{flex} input file consists of three sections, separated by a 451line containing only @samp{%%}. 452 453@cindex format of input file 454@example 455@verbatim 456 definitions 457 %% 458 rules 459 %% 460 user code 461@end verbatim 462@end example 463 464@menu 465* Definitions Section:: 466* Rules Section:: 467* User Code Section:: 468* Comments in the Input:: 469@end menu 470 471@node Definitions Section, Rules Section, Format, Format 472@section Format of the Definitions Section 473 474@cindex input file, Definitions section 475@cindex Definitions, in flex input 476The @dfn{definitions section} contains declarations of simple @dfn{name} 477definitions to simplify the scanner specification, and declarations of 478@dfn{start conditions}, which are explained in a later section. 479 480@cindex aliases, how to define 481@cindex pattern aliases, how to define 482Name definitions have the form: 483 484@example 485@verbatim 486 name definition 487@end verbatim 488@end example 489 490The @samp{name} is a word beginning with a letter or an underscore 491(@samp{_}) followed by zero or more letters, digits, @samp{_}, or 492@samp{-} (dash). The definition is taken to begin at the first 493non-whitespace character following the name and continuing to the end of 494the line. The definition can subsequently be referred to using 495@samp{@{name@}}, which will expand to @samp{(definition)}. For example, 496 497@cindex pattern aliases, defining 498@cindex defining pattern aliases 499@example 500@verbatim 501 DIGIT [0-9] 502 ID [a-z][a-z0-9]* 503@end verbatim 504@end example 505 506Defines @samp{DIGIT} to be a regular expression which matches a single 507digit, and @samp{ID} to be a regular expression which matches a letter 508followed by zero-or-more letters-or-digits. A subsequent reference to 509 510@cindex pattern aliases, use of 511@example 512@verbatim 513 {DIGIT}+"."{DIGIT}* 514@end verbatim 515@end example 516 517is identical to 518 519@example 520@verbatim 521 ([0-9])+"."([0-9])* 522@end verbatim 523@end example 524 525and matches one-or-more digits followed by a @samp{.} followed by 526zero-or-more digits. 527 528@cindex comments in flex input 529An unindented comment (i.e., a line 530beginning with @samp{/*}) is copied verbatim to the output up 531to the next @samp{*/}. 532 533@cindex %@{ and %@}, in Definitions Section 534@cindex embedding C code in flex input 535@cindex C code in flex input 536Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}} 537is also copied verbatim to the output (with the %@{ and %@} symbols 538removed). The %@{ and %@} symbols must appear unindented on lines by 539themselves. 540 541@cindex %top 542 543A @code{%top} block is similar to a @samp{%@{} ... @samp{%@}} block, except 544that the code in a @code{%top} block is relocated to the @emph{top} of the 545generated file, before any flex definitions @footnote{Actually, 546@code{yyIN_HEADER} is defined before the @samp{%top} block.}. 547The @code{%top} block is useful when you want certain preprocessor macros to be 548defined or certain files to be included before the generated code. 549The single characters, @samp{@{} and @samp{@}} are used to delimit the 550@code{%top} block, as show in the example below: 551 552@example 553@verbatim 554 %top{ 555 /* This code goes at the "top" of the generated file. */ 556 #include <stdint.h> 557 #include <inttypes.h> 558 } 559@end verbatim 560@end example 561 562Multiple @code{%top} blocks are allowed, and their order is preserved. 563 564@node Rules Section, User Code Section, Definitions Section, Format 565@section Format of the Rules Section 566 567@cindex input file, Rules Section 568@cindex rules, in flex input 569The @dfn{rules} section of the @code{flex} input contains a series of 570rules of the form: 571 572@example 573@verbatim 574 pattern action 575@end verbatim 576@end example 577 578where the pattern must be unindented and the action must begin 579on the same line. 580@xref{Patterns}, for a further description of patterns and actions. 581 582In the rules section, any indented or %@{ %@} enclosed text appearing 583before the first rule may be used to declare variables which are local 584to the scanning routine and (after the declarations) code which is to be 585executed whenever the scanning routine is entered. Other indented or 586%@{ %@} text in the rule section is still copied to the output, but its 587meaning is not well-defined and it may well cause compile-time errors 588(this feature is present for @acronym{POSIX} compliance. @xref{Lex and 589Posix}, for other such features). 590 591Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}} 592is copied verbatim to the output (with the %@{ and %@} symbols removed). 593The %@{ and %@} symbols must appear unindented on lines by themselves. 594 595@node User Code Section, Comments in the Input, Rules Section, Format 596@section Format of the User Code Section 597 598@cindex input file, user code Section 599@cindex user code, in flex input 600The user code section is simply copied to @file{lex.yy.c} verbatim. It 601is used for companion routines which call or are called by the scanner. 602The presence of this section is optional; if it is missing, the second 603@samp{%%} in the input file may be skipped, too. 604 605@node Comments in the Input, , User Code Section, Format 606@section Comments in the Input 607 608@cindex comments, syntax of 609Flex supports C-style comments, that is, anything between @samp{/*} and 610@samp{*/} is 611considered a comment. Whenever flex encounters a comment, it copies the 612entire comment verbatim to the generated source code. Comments may 613appear just about anywhere, but with the following exceptions: 614 615@itemize 616@cindex comments, in rules section 617@item 618Comments may not appear in the Rules Section wherever flex is expecting 619a regular expression. This means comments may not appear at the 620beginning of a line, or immediately following a list of scanner states. 621@item 622Comments may not appear on an @samp{%option} line in the Definitions 623Section. 624@end itemize 625 626If you want to follow a simple rule, then always begin a comment on a 627new line, with one or more whitespace characters before the initial 628@samp{/*}). This rule will work anywhere in the input file. 629 630All the comments in the following example are valid: 631 632@cindex comments, valid uses of 633@cindex comments in the input 634@example 635@verbatim 636%{ 637/* code block */ 638%} 639 640/* Definitions Section */ 641%x STATE_X 642 643%% 644 /* Rules Section */ 645ruleA /* after regex */ { /* code block */ } /* after code block */ 646 /* Rules Section (indented) */ 647<STATE_X>{ 648ruleC ECHO; 649ruleD ECHO; 650%{ 651/* code block */ 652%} 653} 654%% 655/* User Code Section */ 656 657@end verbatim 658@end example 659 660@node Patterns, Matching, Format, Top 661@chapter Patterns 662 663@cindex patterns, in rules section 664@cindex regular expressions, in patterns 665The patterns in the input (see @ref{Rules Section}) are written using an 666extended set of regular expressions. These are: 667 668@cindex patterns, syntax 669@cindex patterns, syntax 670@table @samp 671@item x 672match the character 'x' 673 674@item . 675any character (byte) except newline 676 677@cindex [] in patterns 678@cindex character classes in patterns, syntax of 679@cindex POSIX, character classes in patterns, syntax of 680@item [xyz] 681a @dfn{character class}; in this case, the pattern 682matches either an 'x', a 'y', or a 'z' 683 684@cindex ranges in patterns 685@item [abj-oZ] 686a "character class" with a range in it; matches 687an 'a', a 'b', any letter from 'j' through 'o', 688or a 'Z' 689 690@cindex ranges in patterns, negating 691@cindex negating ranges in patterns 692@item [^A-Z] 693a "negated character class", i.e., any character 694but those in the class. In this case, any 695character EXCEPT an uppercase letter. 696 697@item [^A-Z\n] 698any character EXCEPT an uppercase letter or 699a newline 700 701@item [a-z]@{-@}[aeiou] 702the lowercase consonants 703 704@item r* 705zero or more r's, where r is any regular expression 706 707@item r+ 708one or more r's 709 710@item r? 711zero or one r's (that is, ``an optional r'') 712 713@cindex braces in patterns 714@item r@{2,5@} 715anywhere from two to five r's 716 717@item r@{2,@} 718two or more r's 719 720@item r@{4@} 721exactly 4 r's 722 723@cindex pattern aliases, expansion of 724@item @{name@} 725the expansion of the @samp{name} definition 726(@pxref{Format}). 727 728@cindex literal text in patterns, syntax of 729@cindex verbatim text in patterns, syntax of 730@item "[xyz]\"foo" 731the literal string: @samp{[xyz]"foo} 732 733@cindex escape sequences in patterns, syntax of 734@item \X 735if X is @samp{a}, @samp{b}, @samp{f}, @samp{n}, @samp{r}, @samp{t}, or 736@samp{v}, then the ANSI-C interpretation of @samp{\x}. Otherwise, a 737literal @samp{X} (used to escape operators such as @samp{*}) 738 739@cindex NULL character in patterns, syntax of 740@item \0 741a NUL character (ASCII code 0) 742 743@cindex octal characters in patterns 744@item \123 745the character with octal value 123 746 747@item \x2a 748the character with hexadecimal value 2a 749 750@item (r) 751match an @samp{r}; parentheses are used to override precedence (see below) 752 753@item (?r-s:pattern) 754apply option @samp{r} and omit option @samp{s} while interpreting pattern. 755Options may be zero or more of the characters @samp{i}, @samp{s}, or @samp{x}. 756 757@samp{i} means case-insensitive. @samp{-i} means case-sensitive. 758 759@samp{s} alters the meaning of the @samp{.} syntax to match any single byte whatsoever. 760@samp{-s} alters the meaning of @samp{.} to match any byte except @samp{\n}. 761 762@samp{x} ignores comments and whitespace in patterns. Whitespace is ignored unless 763it is backslash-escaped, contained within @samp{""}s, or appears inside a 764character class. 765 766The following are all valid: 767 768@verbatim 769(?:foo) same as (foo) 770(?i:ab7) same as ([aA][bB]7) 771(?-i:ab) same as (ab) 772(?s:.) same as [\x00-\xFF] 773(?-s:.) same as [^\n] 774(?ix-s: a . b) same as ([Aa][^\n][bB]) 775(?x:a b) same as ("ab") 776(?x:a\ b) same as ("a b") 777(?x:a" "b) same as ("a b") 778(?x:a[ ]b) same as ("a b") 779(?x:a 780 /* comment */ 781 b 782 c) same as (abc) 783@end verbatim 784 785@item (?# comment ) 786omit everything within @samp{()}. The first @samp{)} 787character encountered ends the pattern. It is not possible to for the comment 788to contain a @samp{)} character. The comment may span lines. 789 790@cindex concatenation, in patterns 791@item rs 792the regular expression @samp{r} followed by the regular expression @samp{s}; called 793@dfn{concatenation} 794 795@item r|s 796either an @samp{r} or an @samp{s} 797 798@cindex trailing context, in patterns 799@item r/s 800an @samp{r} but only if it is followed by an @samp{s}. The text matched by @samp{s} is 801included when determining whether this rule is the longest match, but is 802then returned to the input before the action is executed. So the action 803only sees the text matched by @samp{r}. This type of pattern is called 804@dfn{trailing context}. (There are some combinations of @samp{r/s} that flex 805cannot match correctly. @xref{Limitations}, regarding dangerous trailing 806context.) 807 808@cindex beginning of line, in patterns 809@cindex BOL, in patterns 810@item ^r 811an @samp{r}, but only at the beginning of a line (i.e., 812when just starting to scan, or right after a 813newline has been scanned). 814 815@cindex end of line, in patterns 816@cindex EOL, in patterns 817@item r$ 818an @samp{r}, but only at the end of a line (i.e., just before a 819newline). Equivalent to @samp{r/\n}. 820 821@cindex newline, matching in patterns 822Note that @code{flex}'s notion of ``newline'' is exactly 823whatever the C compiler used to compile @code{flex} 824interprets @samp{\n} as; in particular, on some DOS 825systems you must either filter out @samp{\r}s in the 826input yourself, or explicitly use @samp{r/\r\n} for @samp{r$}. 827 828@cindex start conditions, in patterns 829@item <s>r 830an @samp{r}, but only in start condition @code{s} (see @ref{Start 831Conditions} for discussion of start conditions). 832 833@item <s1,s2,s3>r 834same, but in any of start conditions @code{s1}, @code{s2}, or @code{s3}. 835 836@item <*>r 837an @samp{r} in any start condition, even an exclusive one. 838 839@cindex end of file, in patterns 840@cindex EOF in patterns, syntax of 841@item <<EOF>> 842an end-of-file. 843 844@item <s1,s2><<EOF>> 845an end-of-file when in start condition @code{s1} or @code{s2} 846@end table 847 848Note that inside of a character class, all regular expression operators 849lose their special meaning except escape (@samp{\}) and the character class 850operators, @samp{-}, @samp{]]}, and, at the beginning of the class, @samp{^}. 851 852@cindex patterns, precedence of operators 853The regular expressions listed above are grouped according to 854precedence, from highest precedence at the top to lowest at the bottom. 855Those grouped together have equal precedence (see special note on the 856precedence of the repeat operator, @samp{@{@}}, under the documentation 857for the @samp{--posix} POSIX compliance option). For example, 858 859@cindex patterns, grouping and precedence 860@example 861@verbatim 862 foo|bar* 863@end verbatim 864@end example 865 866is the same as 867 868@example 869@verbatim 870 (foo)|(ba(r*)) 871@end verbatim 872@end example 873 874since the @samp{*} operator has higher precedence than concatenation, 875and concatenation higher than alternation (@samp{|}). This pattern 876therefore matches @emph{either} the string @samp{foo} @emph{or} the 877string @samp{ba} followed by zero-or-more @samp{r}'s. To match 878@samp{foo} or zero-or-more repetitions of the string @samp{bar}, use: 879 880@example 881@verbatim 882 foo|(bar)* 883@end verbatim 884@end example 885 886And to match a sequence of zero or more repetitions of @samp{foo} and 887@samp{bar}: 888 889@cindex patterns, repetitions with grouping 890@example 891@verbatim 892 (foo|bar)* 893@end verbatim 894@end example 895 896@cindex character classes in patterns 897In addition to characters and ranges of characters, character classes 898can also contain @dfn{character class expressions}. These are 899expressions enclosed inside @samp{[}: and @samp{:]} delimiters (which 900themselves must appear between the @samp{[} and @samp{]} of the 901character class. Other elements may occur inside the character class, 902too). The valid expressions are: 903 904@cindex patterns, valid character classes 905@example 906@verbatim 907 [:alnum:] [:alpha:] [:blank:] 908 [:cntrl:] [:digit:] [:graph:] 909 [:lower:] [:print:] [:punct:] 910 [:space:] [:upper:] [:xdigit:] 911@end verbatim 912@end example 913 914These expressions all designate a set of characters equivalent to the 915corresponding standard C @code{isXXX} function. For example, 916@samp{[:alnum:]} designates those characters for which @code{isalnum()} 917returns true - i.e., any alphabetic or numeric character. Some systems 918don't provide @code{isblank()}, so flex defines @samp{[:blank:]} as a 919blank or a tab. 920 921For example, the following character classes are all equivalent: 922 923@cindex character classes, equivalence of 924@cindex patterns, character class equivalence 925@example 926@verbatim 927 [[:alnum:]] 928 [[:alpha:][:digit:]] 929 [[:alpha:][0-9]] 930 [a-zA-Z0-9] 931@end verbatim 932@end example 933 934A word of caution. Character classes are expanded immediately when seen in the @code{flex} input. 935This means the character classes are sensitive to the locale in which @code{flex} 936is executed, and the resulting scanner will not be sensitive to the runtime locale. 937This may or may not be desirable. 938 939 940@itemize 941@cindex case-insensitive, effect on character classes 942@item If your scanner is case-insensitive (the @samp{-i} flag), then 943@samp{[:upper:]} and @samp{[:lower:]} are equivalent to 944@samp{[:alpha:]}. 945 946@anchor{case and character ranges} 947@item Character classes with ranges, such as @samp{[a-Z]}, should be used with 948caution in a case-insensitive scanner if the range spans upper or lowercase 949characters. Flex does not know if you want to fold all upper and lowercase 950characters together, or if you want the literal numeric range specified (with 951no case folding). When in doubt, flex will assume that you meant the literal 952numeric range, and will issue a warning. The exception to this rule is a 953character range such as @samp{[a-z]} or @samp{[S-W]} where it is obvious that you 954want case-folding to occur. Here are some examples with the @samp{-i} flag 955enabled: 956 957@multitable {@samp{[a-zA-Z]}} {ambiguous} {@samp{[A-Z\[\\\]_`a-t]}} {@samp{[@@A-Z\[\\\]_`abc]}} 958@item Range @tab Result @tab Literal Range @tab Alternate Range 959@item @samp{[a-t]} @tab ok @tab @samp{[a-tA-T]} @tab 960@item @samp{[A-T]} @tab ok @tab @samp{[a-tA-T]} @tab 961@item @samp{[A-t]} @tab ambiguous @tab @samp{[A-Z\[\\\]_`a-t]} @tab @samp{[a-tA-T]} 962@item @samp{[_-@{]} @tab ambiguous @tab @samp{[_`a-z@{]} @tab @samp{[_`a-zA-Z@{]} 963@item @samp{[@@-C]} @tab ambiguous @tab @samp{[@@ABC]} @tab @samp{[@@A-Z\[\\\]_`abc]} 964@end multitable 965 966@cindex end of line, in negated character classes 967@cindex EOL, in negated character classes 968@item 969A negated character class such as the example @samp{[^A-Z]} above 970@emph{will} match a newline unless @samp{\n} (or an equivalent escape 971sequence) is one of the characters explicitly present in the negated 972character class (e.g., @samp{[^A-Z\n]}). This is unlike how many other 973regular expression tools treat negated character classes, but 974unfortunately the inconsistency is historically entrenched. Matching 975newlines means that a pattern like @samp{[^"]*} can match the entire 976input unless there's another quote in the input. 977 978Flex allows negation of character class expressions by prepending @samp{^} to 979the POSIX character class name. 980 981@example 982@verbatim 983 [:^alnum:] [:^alpha:] [:^blank:] 984 [:^cntrl:] [:^digit:] [:^graph:] 985 [:^lower:] [:^print:] [:^punct:] 986 [:^space:] [:^upper:] [:^xdigit:] 987@end verbatim 988@end example 989 990Flex will issue a warning if the expressions @samp{[:^upper:]} and 991@samp{[:^lower:]} appear in a case-insensitive scanner, since their meaning is 992unclear. The current behavior is to skip them entirely, but this may change 993without notice in future revisions of flex. 994 995@item 996 997The @samp{@{-@}} operator computes the difference of two character classes. For 998example, @samp{[a-c]@{-@}[b-z]} represents all the characters in the class 999@samp{[a-c]} that are not in the class @samp{[b-z]} (which in this case, is 1000just the single character @samp{a}). The @samp{@{-@}} operator is left 1001associative, so @samp{[abc]@{-@}[b]@{-@}[c]} is the same as @samp{[a]}. Be careful 1002not to accidentally create an empty set, which will never match. 1003 1004@item 1005 1006The @samp{@{+@}} operator computes the union of two character classes. For 1007example, @samp{[a-z]@{+@}[0-9]} is the same as @samp{[a-z0-9]}. This operator 1008is useful when preceded by the result of a difference operation, as in, 1009@samp{[[:alpha:]]@{-@}[[:lower:]]@{+@}[q]}, which is equivalent to 1010@samp{[A-Zq]} in the "C" locale. 1011 1012@cindex trailing context, limits of 1013@cindex ^ as non-special character in patterns 1014@cindex $ as normal character in patterns 1015@item 1016A rule can have at most one instance of trailing context (the @samp{/} operator 1017or the @samp{$} operator). The start condition, @samp{^}, and @samp{<<EOF>>} patterns 1018can only occur at the beginning of a pattern, and, as well as with @samp{/} and @samp{$}, 1019cannot be grouped inside parentheses. A @samp{^} which does not occur at 1020the beginning of a rule or a @samp{$} which does not occur at the end of 1021a rule loses its special properties and is treated as a normal character. 1022 1023@item 1024The following are invalid: 1025 1026@cindex patterns, invalid trailing context 1027@example 1028@verbatim 1029 foo/bar$ 1030 <sc1>foo<sc2>bar 1031@end verbatim 1032@end example 1033 1034Note that the first of these can be written @samp{foo/bar\n}. 1035 1036@item 1037The following will result in @samp{$} or @samp{^} being treated as a normal character: 1038 1039@cindex patterns, special characters treated as non-special 1040@example 1041@verbatim 1042 foo|(bar$) 1043 foo|^bar 1044@end verbatim 1045@end example 1046 1047If the desired meaning is a @samp{foo} or a 1048@samp{bar}-followed-by-a-newline, the following could be used (the 1049special @code{|} action is explained below, @pxref{Actions}): 1050 1051@cindex patterns, end of line 1052@example 1053@verbatim 1054 foo | 1055 bar$ /* action goes here */ 1056@end verbatim 1057@end example 1058 1059A similar trick will work for matching a @samp{foo} or a 1060@samp{bar}-at-the-beginning-of-a-line. 1061@end itemize 1062 1063@node Matching, Actions, Patterns, Top 1064@chapter How the Input Is Matched 1065 1066@cindex patterns, matching 1067@cindex input, matching 1068@cindex trailing context, matching 1069@cindex matching, and trailing context 1070@cindex matching, length of 1071@cindex matching, multiple matches 1072When the generated scanner is run, it analyzes its input looking for 1073strings which match any of its patterns. If it finds more than one 1074match, it takes the one matching the most text (for trailing context 1075rules, this includes the length of the trailing part, even though it 1076will then be returned to the input). If it finds two or more matches of 1077the same length, the rule listed first in the @code{flex} input file is 1078chosen. 1079 1080@cindex token 1081@cindex yytext 1082@cindex yyleng 1083Once the match is determined, the text corresponding to the match 1084(called the @dfn{token}) is made available in the global character 1085pointer @code{yytext}, and its length in the global integer 1086@code{yyleng}. The @dfn{action} corresponding to the matched pattern is 1087then executed (@pxref{Actions}), and then the remaining input is scanned 1088for another match. 1089 1090@cindex default rule 1091If no match is found, then the @dfn{default rule} is executed: the next 1092character in the input is considered matched and copied to the standard 1093output. Thus, the simplest valid @code{flex} input is: 1094 1095@cindex minimal scanner 1096@example 1097@verbatim 1098 %% 1099@end verbatim 1100@end example 1101 1102which generates a scanner that simply copies its input (one character at 1103a time) to its output. 1104 1105@cindex yytext, two types of 1106@cindex %array, use of 1107@cindex %pointer, use of 1108@vindex yytext 1109Note that @code{yytext} can be defined in two different ways: either as 1110a character @emph{pointer} or as a character @emph{array}. You can 1111control which definition @code{flex} uses by including one of the 1112special directives @code{%pointer} or @code{%array} in the first 1113(definitions) section of your flex input. The default is 1114@code{%pointer}, unless you use the @samp{-l} lex compatibility option, 1115in which case @code{yytext} will be an array. The advantage of using 1116@code{%pointer} is substantially faster scanning and no buffer overflow 1117when matching very large tokens (unless you run out of dynamic memory). 1118The disadvantage is that you are restricted in how your actions can 1119modify @code{yytext} (@pxref{Actions}), and calls to the @code{unput()} 1120function destroys the present contents of @code{yytext}, which can be a 1121considerable porting headache when moving between different @code{lex} 1122versions. 1123 1124@cindex %array, advantages of 1125The advantage of @code{%array} is that you can then modify @code{yytext} 1126to your heart's content, and calls to @code{unput()} do not destroy 1127@code{yytext} (@pxref{Actions}). Furthermore, existing @code{lex} 1128programs sometimes access @code{yytext} externally using declarations of 1129the form: 1130 1131@example 1132@verbatim 1133 extern char yytext[]; 1134@end verbatim 1135@end example 1136 1137This definition is erroneous when used with @code{%pointer}, but correct 1138for @code{%array}. 1139 1140The @code{%array} declaration defines @code{yytext} to be an array of 1141@code{YYLMAX} characters, which defaults to a fairly large value. You 1142can change the size by simply #define'ing @code{YYLMAX} to a different 1143value in the first section of your @code{flex} input. As mentioned 1144above, with @code{%pointer} yytext grows dynamically to accommodate 1145large tokens. While this means your @code{%pointer} scanner can 1146accommodate very large tokens (such as matching entire blocks of 1147comments), bear in mind that each time the scanner must resize 1148@code{yytext} it also must rescan the entire token from the beginning, 1149so matching such tokens can prove slow. @code{yytext} presently does 1150@emph{not} dynamically grow if a call to @code{unput()} results in too 1151much text being pushed back; instead, a run-time error results. 1152 1153@cindex %array, with C++ 1154Also note that you cannot use @code{%array} with C++ scanner classes 1155(@pxref{Cxx}). 1156 1157@node Actions, Generated Scanner, Matching, Top 1158@chapter Actions 1159 1160@cindex actions 1161Each pattern in a rule has a corresponding @dfn{action}, which can be 1162any arbitrary C statement. The pattern ends at the first non-escaped 1163whitespace character; the remainder of the line is its action. If the 1164action is empty, then when the pattern is matched the input token is 1165simply discarded. For example, here is the specification for a program 1166which deletes all occurrences of @samp{zap me} from its input: 1167 1168@cindex deleting lines from input 1169@example 1170@verbatim 1171 %% 1172 "zap me" 1173@end verbatim 1174@end example 1175 1176This example will copy all other characters in the input to the output 1177since they will be matched by the default rule. 1178 1179Here is a program which compresses multiple blanks and tabs down to a 1180single blank, and throws away whitespace found at the end of a line: 1181 1182@cindex whitespace, compressing 1183@cindex compressing whitespace 1184@example 1185@verbatim 1186 %% 1187 [ \t]+ putchar( ' ' ); 1188 [ \t]+$ /* ignore this token */ 1189@end verbatim 1190@end example 1191 1192@cindex %@{ and %@}, in Rules Section 1193@cindex actions, use of @{ and @} 1194@cindex actions, embedded C strings 1195@cindex C-strings, in actions 1196@cindex comments, in actions 1197If the action contains a @samp{@{}, then the action spans till the 1198balancing @samp{@}} is found, and the action may cross multiple lines. 1199@code{flex} knows about C strings and comments and won't be fooled by 1200braces found within them, but also allows actions to begin with 1201@samp{%@{} and will consider the action to be all the text up to the 1202next @samp{%@}} (regardless of ordinary braces inside the action). 1203 1204@cindex |, in actions 1205An action consisting solely of a vertical bar (@samp{|}) means ``same as the 1206action for the next rule''. See below for an illustration. 1207 1208Actions can include arbitrary C code, including @code{return} statements 1209to return a value to whatever routine called @code{yylex()}. Each time 1210@code{yylex()} is called it continues processing tokens from where it 1211last left off until it either reaches the end of the file or executes a 1212return. 1213 1214@cindex yytext, modification of 1215Actions are free to modify @code{yytext} except for lengthening it 1216(adding characters to its end--these will overwrite later characters in 1217the input stream). This however does not apply when using @code{%array} 1218(@pxref{Matching}). In that case, @code{yytext} may be freely modified 1219in any way. 1220 1221@cindex yyleng, modification of 1222@cindex yymore, and yyleng 1223Actions are free to modify @code{yyleng} except they should not do so if 1224the action also includes use of @code{yymore()} (see below). 1225 1226@cindex preprocessor macros, for use in actions 1227There are a number of special directives which can be included within an 1228action: 1229 1230@table @code 1231@item ECHO 1232@cindex ECHO 1233copies yytext to the scanner's output. 1234 1235@item BEGIN 1236@cindex BEGIN 1237followed by the name of a start condition places the scanner in the 1238corresponding start condition (see below). 1239 1240@item REJECT 1241@cindex REJECT 1242directs the scanner to proceed on to the ``second best'' rule which 1243matched the input (or a prefix of the input). The rule is chosen as 1244described above in @ref{Matching}, and @code{yytext} and @code{yyleng} 1245set up appropriately. It may either be one which matched as much text 1246as the originally chosen rule but came later in the @code{flex} input 1247file, or one which matched less text. For example, the following will 1248both count the words in the input and call the routine @code{special()} 1249whenever @samp{frob} is seen: 1250 1251@example 1252@verbatim 1253 int word_count = 0; 1254 %% 1255 1256 frob special(); REJECT; 1257 [^ \t\n]+ ++word_count; 1258@end verbatim 1259@end example 1260 1261Without the @code{REJECT}, any occurrences of @samp{frob} in the input 1262would not be counted as words, since the scanner normally executes only 1263one action per token. Multiple uses of @code{REJECT} are allowed, each 1264one finding the next best choice to the currently active rule. For 1265example, when the following scanner scans the token @samp{abcd}, it will 1266write @samp{abcdabcaba} to the output: 1267 1268@cindex REJECT, calling multiple times 1269@cindex |, use of 1270@example 1271@verbatim 1272 %% 1273 a | 1274 ab | 1275 abc | 1276 abcd ECHO; REJECT; 1277 .|\n /* eat up any unmatched character */ 1278@end verbatim 1279@end example 1280 1281The first three rules share the fourth's action since they use the 1282special @samp{|} action. 1283 1284@code{REJECT} is a particularly expensive feature in terms of scanner 1285performance; if it is used in @emph{any} of the scanner's actions it 1286will slow down @emph{all} of the scanner's matching. Furthermore, 1287@code{REJECT} cannot be used with the @samp{-Cf} or @samp{-CF} options 1288(@pxref{Scanner Options}). 1289 1290Note also that unlike the other special actions, @code{REJECT} is a 1291@emph{branch}. Code immediately following it in the action will 1292@emph{not} be executed. 1293 1294@item yymore() 1295@cindex yymore() 1296tells the scanner that the next time it matches a rule, the 1297corresponding token should be @emph{appended} onto the current value of 1298@code{yytext} rather than replacing it. For example, given the input 1299@samp{mega-kludge} the following will write @samp{mega-mega-kludge} to 1300the output: 1301 1302@cindex yymore(), mega-kludge 1303@cindex yymore() to append token to previous token 1304@example 1305@verbatim 1306 %% 1307 mega- ECHO; yymore(); 1308 kludge ECHO; 1309@end verbatim 1310@end example 1311 1312First @samp{mega-} is matched and echoed to the output. Then @samp{kludge} 1313is matched, but the previous @samp{mega-} is still hanging around at the 1314beginning of 1315@code{yytext} 1316so the 1317@code{ECHO} 1318for the @samp{kludge} rule will actually write @samp{mega-kludge}. 1319@end table 1320 1321@cindex yymore, performance penalty of 1322Two notes regarding use of @code{yymore()}. First, @code{yymore()} 1323depends on the value of @code{yyleng} correctly reflecting the size of 1324the current token, so you must not modify @code{yyleng} if you are using 1325@code{yymore()}. Second, the presence of @code{yymore()} in the 1326scanner's action entails a minor performance penalty in the scanner's 1327matching speed. 1328 1329@cindex yyless() 1330@code{yyless(n)} returns all but the first @code{n} characters of the 1331current token back to the input stream, where they will be rescanned 1332when the scanner looks for the next match. @code{yytext} and 1333@code{yyleng} are adjusted appropriately (e.g., @code{yyleng} will now 1334be equal to @code{n}). For example, on the input @samp{foobar} the 1335following will write out @samp{foobarbar}: 1336 1337@cindex yyless(), pushing back characters 1338@cindex pushing back characters with yyless 1339@example 1340@verbatim 1341 %% 1342 foobar ECHO; yyless(3); 1343 [a-z]+ ECHO; 1344@end verbatim 1345@end example 1346 1347An argument of 0 to @code{yyless()} will cause the entire current input 1348string to be scanned again. Unless you've changed how the scanner will 1349subsequently process its input (using @code{BEGIN}, for example), this 1350will result in an endless loop. 1351 1352Note that @code{yyless()} is a macro and can only be used in the flex 1353input file, not from other source files. 1354 1355@cindex unput() 1356@cindex pushing back characters with unput 1357@code{unput(c)} puts the character @code{c} back onto the input stream. 1358It will be the next character scanned. The following action will take 1359the current token and cause it to be rescanned enclosed in parentheses. 1360 1361@cindex unput(), pushing back characters 1362@cindex pushing back characters with unput() 1363@example 1364@verbatim 1365 { 1366 int i; 1367 /* Copy yytext because unput() trashes yytext */ 1368 char *yycopy = strdup( yytext ); 1369 unput( ')' ); 1370 for ( i = yyleng - 1; i >= 0; --i ) 1371 unput( yycopy[i] ); 1372 unput( '(' ); 1373 free( yycopy ); 1374 } 1375@end verbatim 1376@end example 1377 1378Note that since each @code{unput()} puts the given character back at the 1379@emph{beginning} of the input stream, pushing back strings must be done 1380back-to-front. 1381 1382@cindex %pointer, and unput() 1383@cindex unput(), and %pointer 1384An important potential problem when using @code{unput()} is that if you 1385are using @code{%pointer} (the default), a call to @code{unput()} 1386@emph{destroys} the contents of @code{yytext}, starting with its 1387rightmost character and devouring one character to the left with each 1388call. If you need the value of @code{yytext} preserved after a call to 1389@code{unput()} (as in the above example), you must either first copy it 1390elsewhere, or build your scanner using @code{%array} instead 1391(@pxref{Matching}). 1392 1393@cindex pushing back EOF 1394@cindex EOF, pushing back 1395Finally, note that you cannot put back @samp{EOF} to attempt to mark the 1396input stream with an end-of-file. 1397 1398@cindex input() 1399@code{input()} reads the next character from the input stream. For 1400example, the following is one way to eat up C comments: 1401 1402@cindex comments, discarding 1403@cindex discarding C comments 1404@example 1405@verbatim 1406 %% 1407 "/*" { 1408 register int c; 1409 1410 for ( ; ; ) 1411 { 1412 while ( (c = input()) != '*' && 1413 c != EOF ) 1414 ; /* eat up text of comment */ 1415 1416 if ( c == '*' ) 1417 { 1418 while ( (c = input()) == '*' ) 1419 ; 1420 if ( c == '/' ) 1421 break; /* found the end */ 1422 } 1423 1424 if ( c == EOF ) 1425 { 1426 error( "EOF in comment" ); 1427 break; 1428 } 1429 } 1430 } 1431@end verbatim 1432@end example 1433 1434@cindex input(), and C++ 1435@cindex yyinput() 1436(Note that if the scanner is compiled using @code{C++}, then 1437@code{input()} is instead referred to as @b{yyinput()}, in order to 1438avoid a name clash with the @code{C++} stream by the name of 1439@code{input}.) 1440 1441@cindex flushing the internal buffer 1442@cindex YY_FLUSH_BUFFER 1443@code{YY_FLUSH_BUFFER;} flushes the scanner's internal buffer so that 1444the next time the scanner attempts to match a token, it will first 1445refill the buffer using @code{YY_INPUT()} (@pxref{Generated Scanner}). 1446This action is a special case of the more general 1447@code{yy_flush_buffer;} function, described below (@pxref{Multiple 1448Input Buffers}) 1449 1450@cindex yyterminate() 1451@cindex terminating with yyterminate() 1452@cindex exiting with yyterminate() 1453@cindex halting with yyterminate() 1454@code{yyterminate()} can be used in lieu of a return statement in an 1455action. It terminates the scanner and returns a 0 to the scanner's 1456caller, indicating ``all done''. By default, @code{yyterminate()} is 1457also called when an end-of-file is encountered. It is a macro and may 1458be redefined. 1459 1460@node Generated Scanner, Start Conditions, Actions, Top 1461@chapter The Generated Scanner 1462 1463@cindex yylex(), in generated scanner 1464The output of @code{flex} is the file @file{lex.yy.c}, which contains 1465the scanning routine @code{yylex()}, a number of tables used by it for 1466matching tokens, and a number of auxiliary routines and macros. By 1467default, @code{yylex()} is declared as follows: 1468 1469@example 1470@verbatim 1471 int yylex() 1472 { 1473 ... various definitions and the actions in here ... 1474 } 1475@end verbatim 1476@end example 1477 1478@cindex yylex(), overriding 1479(If your environment supports function prototypes, then it will be 1480@code{int yylex( void )}.) This definition may be changed by defining 1481the @code{YY_DECL} macro. For example, you could use: 1482 1483@cindex yylex, overriding the prototype of 1484@example 1485@verbatim 1486 #define YY_DECL float lexscan( a, b ) float a, b; 1487@end verbatim 1488@end example 1489 1490to give the scanning routine the name @code{lexscan}, returning a float, 1491and taking two floats as arguments. Note that if you give arguments to 1492the scanning routine using a K&R-style/non-prototyped function 1493declaration, you must terminate the definition with a semi-colon (;). 1494 1495@code{flex} generates @samp{C99} function definitions by 1496default. However flex does have the ability to generate obsolete, er, 1497@samp{traditional}, function definitions. This is to support 1498bootstrapping gcc on old systems. Unfortunately, traditional 1499definitions prevent us from using any standard data types smaller than 1500int (such as short, char, or bool) as function arguments. For this 1501reason, future versions of @code{flex} may generate standard C99 code 1502only, leaving K&R-style functions to the historians. Currently, if you 1503do @strong{not} want @samp{C99} definitions, then you must use 1504@code{%option noansi-definitions}. 1505 1506@cindex stdin, default for yyin 1507@cindex yyin 1508Whenever @code{yylex()} is called, it scans tokens from the global input 1509file @file{yyin} (which defaults to stdin). It continues until it 1510either reaches an end-of-file (at which point it returns the value 0) or 1511one of its actions executes a @code{return} statement. 1512 1513@cindex EOF and yyrestart() 1514@cindex end-of-file, and yyrestart() 1515@cindex yyrestart() 1516If the scanner reaches an end-of-file, subsequent calls are undefined 1517unless either @file{yyin} is pointed at a new input file (in which case 1518scanning continues from that file), or @code{yyrestart()} is called. 1519@code{yyrestart()} takes one argument, a @code{FILE *} pointer (which 1520can be NULL, if you've set up @code{YY_INPUT} to scan from a source other 1521than @code{yyin}), and initializes @file{yyin} for scanning from that 1522file. Essentially there is no difference between just assigning 1523@file{yyin} to a new input file or using @code{yyrestart()} to do so; 1524the latter is available for compatibility with previous versions of 1525@code{flex}, and because it can be used to switch input files in the 1526middle of scanning. It can also be used to throw away the current input 1527buffer, by calling it with an argument of @file{yyin}; but it would be 1528better to use @code{YY_FLUSH_BUFFER} (@pxref{Actions}). Note that 1529@code{yyrestart()} does @emph{not} reset the start condition to 1530@code{INITIAL} (@pxref{Start Conditions}). 1531 1532@cindex RETURN, within actions 1533If @code{yylex()} stops scanning due to executing a @code{return} 1534statement in one of the actions, the scanner may then be called again 1535and it will resume scanning where it left off. 1536 1537@cindex YY_INPUT 1538By default (and for purposes of efficiency), the scanner uses 1539block-reads rather than simple @code{getc()} calls to read characters 1540from @file{yyin}. The nature of how it gets its input can be controlled 1541by defining the @code{YY_INPUT} macro. The calling sequence for 1542@code{YY_INPUT()} is @code{YY_INPUT(buf,result,max_size)}. Its action 1543is to place up to @code{max_size} characters in the character array 1544@code{buf} and return in the integer variable @code{result} either the 1545number of characters read or the constant @code{YY_NULL} (0 on Unix 1546systems) to indicate @samp{EOF}. The default @code{YY_INPUT} reads from 1547the global file-pointer @file{yyin}. 1548 1549@cindex YY_INPUT, overriding 1550Here is a sample definition of @code{YY_INPUT} (in the definitions 1551section of the input file): 1552 1553@example 1554@verbatim 1555 %{ 1556 #define YY_INPUT(buf,result,max_size) \ 1557 { \ 1558 int c = getchar(); \ 1559 result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \ 1560 } 1561 %} 1562@end verbatim 1563@end example 1564 1565This definition will change the input processing to occur one character 1566at a time. 1567 1568@cindex yywrap() 1569When the scanner receives an end-of-file indication from YY_INPUT, it 1570then checks the @code{yywrap()} function. If @code{yywrap()} returns 1571false (zero), then it is assumed that the function has gone ahead and 1572set up @file{yyin} to point to another input file, and scanning 1573continues. If it returns true (non-zero), then the scanner terminates, 1574returning 0 to its caller. Note that in either case, the start 1575condition remains unchanged; it does @emph{not} revert to 1576@code{INITIAL}. 1577 1578@cindex yywrap, default for 1579@cindex noyywrap, %option 1580@cindex %option noyywrapp 1581If you do not supply your own version of @code{yywrap()}, then you must 1582either use @code{%option noyywrap} (in which case the scanner behaves as 1583though @code{yywrap()} returned 1), or you must link with @samp{-lfl} to 1584obtain the default version of the routine, which always returns 1. 1585 1586For scanning from in-memory buffers (e.g., scanning strings), see 1587@ref{Scanning Strings}. @xref{Multiple Input Buffers}. 1588 1589@cindex ECHO, and yyout 1590@cindex yyout 1591@cindex stdout, as default for yyout 1592The scanner writes its @code{ECHO} output to the @file{yyout} global 1593(default, @file{stdout}), which may be redefined by the user simply by 1594assigning it to some other @code{FILE} pointer. 1595 1596@node Start Conditions, Multiple Input Buffers, Generated Scanner, Top 1597@chapter Start Conditions 1598 1599@cindex start conditions 1600@code{flex} provides a mechanism for conditionally activating rules. 1601Any rule whose pattern is prefixed with @samp{<sc>} will only be active 1602when the scanner is in the @dfn{start condition} named @code{sc}. For 1603example, 1604 1605@c proofread edit stopped here 1606@example 1607@verbatim 1608 <STRING>[^"]* { /* eat up the string body ... */ 1609 ... 1610 } 1611@end verbatim 1612@end example 1613 1614will be active only when the scanner is in the @code{STRING} start 1615condition, and 1616 1617@cindex start conditions, multiple 1618@example 1619@verbatim 1620 <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */ 1621 ... 1622 } 1623@end verbatim 1624@end example 1625 1626will be active only when the current start condition is either 1627@code{INITIAL}, @code{STRING}, or @code{QUOTE}. 1628 1629@cindex start conditions, inclusive v.s.@: exclusive 1630Start conditions are declared in the definitions (first) section of the 1631input using unindented lines beginning with either @samp{%s} or 1632@samp{%x} followed by a list of names. The former declares 1633@dfn{inclusive} start conditions, the latter @dfn{exclusive} start 1634conditions. A start condition is activated using the @code{BEGIN} 1635action. Until the next @code{BEGIN} action is executed, rules with the 1636given start condition will be active and rules with other start 1637conditions will be inactive. If the start condition is inclusive, then 1638rules with no start conditions at all will also be active. If it is 1639exclusive, then @emph{only} rules qualified with the start condition 1640will be active. A set of rules contingent on the same exclusive start 1641condition describe a scanner which is independent of any of the other 1642rules in the @code{flex} input. Because of this, exclusive start 1643conditions make it easy to specify ``mini-scanners'' which scan portions 1644of the input that are syntactically different from the rest (e.g., 1645comments). 1646 1647If the distinction between inclusive and exclusive start conditions 1648is still a little vague, here's a simple example illustrating the 1649connection between the two. The set of rules: 1650 1651@cindex start conditions, inclusive 1652@example 1653@verbatim 1654 %s example 1655 %% 1656 1657 <example>foo do_something(); 1658 1659 bar something_else(); 1660@end verbatim 1661@end example 1662 1663is equivalent to 1664 1665@cindex start conditions, exclusive 1666@example 1667@verbatim 1668 %x example 1669 %% 1670 1671 <example>foo do_something(); 1672 1673 <INITIAL,example>bar something_else(); 1674@end verbatim 1675@end example 1676 1677Without the @code{<INITIAL,example>} qualifier, the @code{bar} pattern in 1678the second example wouldn't be active (i.e., couldn't match) when in 1679start condition @code{example}. If we just used @code{<example>} to 1680qualify @code{bar}, though, then it would only be active in 1681@code{example} and not in @code{INITIAL}, while in the first example 1682it's active in both, because in the first example the @code{example} 1683start condition is an inclusive @code{(%s)} start condition. 1684 1685@cindex start conditions, special wildcard condition 1686Also note that the special start-condition specifier 1687@code{<*>} 1688matches every start condition. Thus, the above example could also 1689have been written: 1690 1691@cindex start conditions, use of wildcard condition (<*>) 1692@example 1693@verbatim 1694 %x example 1695 %% 1696 1697 <example>foo do_something(); 1698 1699 <*>bar something_else(); 1700@end verbatim 1701@end example 1702 1703The default rule (to @code{ECHO} any unmatched character) remains active 1704in start conditions. It is equivalent to: 1705 1706@cindex start conditions, behavior of default rule 1707@example 1708@verbatim 1709 <*>.|\n ECHO; 1710@end verbatim 1711@end example 1712 1713@cindex BEGIN, explanation 1714@findex BEGIN 1715@vindex INITIAL 1716@code{BEGIN(0)} returns to the original state where only the rules with 1717no start conditions are active. This state can also be referred to as 1718the start-condition @code{INITIAL}, so @code{BEGIN(INITIAL)} is 1719equivalent to @code{BEGIN(0)}. (The parentheses around the start 1720condition name are not required but are considered good style.) 1721 1722@code{BEGIN} actions can also be given as indented code at the beginning 1723of the rules section. For example, the following will cause the scanner 1724to enter the @code{SPECIAL} start condition whenever @code{yylex()} is 1725called and the global variable @code{enter_special} is true: 1726 1727@cindex start conditions, using BEGIN 1728@example 1729@verbatim 1730 int enter_special; 1731 1732 %x SPECIAL 1733 %% 1734 if ( enter_special ) 1735 BEGIN(SPECIAL); 1736 1737 <SPECIAL>blahblahblah 1738 ...more rules follow... 1739@end verbatim 1740@end example 1741 1742To illustrate the uses of start conditions, here is a scanner which 1743provides two different interpretations of a string like @samp{123.456}. 1744By default it will treat it as three tokens, the integer @samp{123}, a 1745dot (@samp{.}), and the integer @samp{456}. But if the string is 1746preceded earlier in the line by the string @samp{expect-floats} it will 1747treat it as a single token, the floating-point number @samp{123.456}: 1748 1749@cindex start conditions, for different interpretations of same input 1750@example 1751@verbatim 1752 %{ 1753 #include <math.h> 1754 %} 1755 %s expect 1756 1757 %% 1758 expect-floats BEGIN(expect); 1759 1760 <expect>[0-9]+.[0-9]+ { 1761 printf( "found a float, = %f\n", 1762 atof( yytext ) ); 1763 } 1764 <expect>\n { 1765 /* that's the end of the line, so 1766 * we need another "expect-number" 1767 * before we'll recognize any more 1768 * numbers 1769 */ 1770 BEGIN(INITIAL); 1771 } 1772 1773 [0-9]+ { 1774 printf( "found an integer, = %d\n", 1775 atoi( yytext ) ); 1776 } 1777 1778 "." printf( "found a dot\n" ); 1779@end verbatim 1780@end example 1781 1782@cindex comments, example of scanning C comments 1783Here is a scanner which recognizes (and discards) C comments while 1784maintaining a count of the current input line. 1785 1786@cindex recognizing C comments 1787@example 1788@verbatim 1789 %x comment 1790 %% 1791 int line_num = 1; 1792 1793 "/*" BEGIN(comment); 1794 1795 <comment>[^*\n]* /* eat anything that's not a '*' */ 1796 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ 1797 <comment>\n ++line_num; 1798 <comment>"*"+"/" BEGIN(INITIAL); 1799@end verbatim 1800@end example 1801 1802This scanner goes to a bit of trouble to match as much 1803text as possible with each rule. In general, when attempting to write 1804a high-speed scanner try to match as much possible in each rule, as 1805it's a big win. 1806 1807Note that start-conditions names are really integer values and 1808can be stored as such. Thus, the above could be extended in the 1809following fashion: 1810 1811@cindex start conditions, integer values 1812@cindex using integer values of start condition names 1813@example 1814@verbatim 1815 %x comment foo 1816 %% 1817 int line_num = 1; 1818 int comment_caller; 1819 1820 "/*" { 1821 comment_caller = INITIAL; 1822 BEGIN(comment); 1823 } 1824 1825 ... 1826 1827 <foo>"/*" { 1828 comment_caller = foo; 1829 BEGIN(comment); 1830 } 1831 1832 <comment>[^*\n]* /* eat anything that's not a '*' */ 1833 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ 1834 <comment>\n ++line_num; 1835 <comment>"*"+"/" BEGIN(comment_caller); 1836@end verbatim 1837@end example 1838 1839@cindex YY_START, example 1840Furthermore, you can access the current start condition using the 1841integer-valued @code{YY_START} macro. For example, the above 1842assignments to @code{comment_caller} could instead be written 1843 1844@cindex getting current start state with YY_START 1845@example 1846@verbatim 1847 comment_caller = YY_START; 1848@end verbatim 1849@end example 1850 1851@vindex YY_START 1852Flex provides @code{YYSTATE} as an alias for @code{YY_START} (since that 1853is what's used by AT&T @code{lex}). 1854 1855For historical reasons, start conditions do not have their own 1856name-space within the generated scanner. The start condition names are 1857unmodified in the generated scanner and generated header. 1858@xref{option-header}. @xref{option-prefix}. 1859 1860 1861 1862Finally, here's an example of how to match C-style quoted strings using 1863exclusive start conditions, including expanded escape sequences (but 1864not including checking for a string that's too long): 1865 1866@cindex matching C-style double-quoted strings 1867@example 1868@verbatim 1869 %x str 1870 1871 %% 1872 char string_buf[MAX_STR_CONST]; 1873 char *string_buf_ptr; 1874 1875 1876 \" string_buf_ptr = string_buf; BEGIN(str); 1877 1878 <str>\" { /* saw closing quote - all done */ 1879 BEGIN(INITIAL); 1880 *string_buf_ptr = '\0'; 1881 /* return string constant token type and 1882 * value to parser 1883 */ 1884 } 1885 1886 <str>\n { 1887 /* error - unterminated string constant */ 1888 /* generate error message */ 1889 } 1890 1891 <str>\\[0-7]{1,3} { 1892 /* octal escape sequence */ 1893 int result; 1894 1895 (void) sscanf( yytext + 1, "%o", &result ); 1896 1897 if ( result > 0xff ) 1898 /* error, constant is out-of-bounds */ 1899 1900 *string_buf_ptr++ = result; 1901 } 1902 1903 <str>\\[0-9]+ { 1904 /* generate error - bad escape sequence; something 1905 * like '\48' or '\0777777' 1906 */ 1907 } 1908 1909 <str>\\n *string_buf_ptr++ = '\n'; 1910 <str>\\t *string_buf_ptr++ = '\t'; 1911 <str>\\r *string_buf_ptr++ = '\r'; 1912 <str>\\b *string_buf_ptr++ = '\b'; 1913 <str>\\f *string_buf_ptr++ = '\f'; 1914 1915 <str>\\(.|\n) *string_buf_ptr++ = yytext[1]; 1916 1917 <str>[^\\\n\"]+ { 1918 char *yptr = yytext; 1919 1920 while ( *yptr ) 1921 *string_buf_ptr++ = *yptr++; 1922 } 1923@end verbatim 1924@end example 1925 1926@cindex start condition, applying to multiple patterns 1927Often, such as in some of the examples above, you wind up writing a 1928whole bunch of rules all preceded by the same start condition(s). Flex 1929makes this a little easier and cleaner by introducing a notion of start 1930condition @dfn{scope}. A start condition scope is begun with: 1931 1932@example 1933@verbatim 1934 <SCs>{ 1935@end verbatim 1936@end example 1937 1938where @code{SCs} is a list of one or more start conditions. Inside the 1939start condition scope, every rule automatically has the prefix 1940@code{SCs>} applied to it, until a @samp{@}} which matches the initial 1941@samp{@{}. So, for example, 1942 1943@cindex extended scope of start conditions 1944@example 1945@verbatim 1946 <ESC>{ 1947 "\\n" return '\n'; 1948 "\\r" return '\r'; 1949 "\\f" return '\f'; 1950 "\\0" return '\0'; 1951 } 1952@end verbatim 1953@end example 1954 1955is equivalent to: 1956 1957@example 1958@verbatim 1959 <ESC>"\\n" return '\n'; 1960 <ESC>"\\r" return '\r'; 1961 <ESC>"\\f" return '\f'; 1962 <ESC>"\\0" return '\0'; 1963@end verbatim 1964@end example 1965 1966Start condition scopes may be nested. 1967 1968@cindex stacks, routines for manipulating 1969@cindex start conditions, use of a stack 1970 1971The following routines are available for manipulating stacks of start conditions: 1972 1973@deftypefun void yy_push_state ( int @code{new_state} ) 1974pushes the current start condition onto the top of the start condition 1975stack and switches to 1976@code{new_state} 1977as though you had used 1978@code{BEGIN new_state} 1979(recall that start condition names are also integers). 1980@end deftypefun 1981 1982@deftypefun void yy_pop_state () 1983pops the top of the stack and switches to it via 1984@code{BEGIN}. 1985@end deftypefun 1986 1987@deftypefun int yy_top_state () 1988returns the top of the stack without altering the stack's contents. 1989@end deftypefun 1990 1991@cindex memory, for start condition stacks 1992The start condition stack grows dynamically and so has no built-in size 1993limitation. If memory is exhausted, program execution aborts. 1994 1995To use start condition stacks, your scanner must include a @code{%option 1996stack} directive (@pxref{Scanner Options}). 1997 1998@node Multiple Input Buffers, EOF, Start Conditions, Top 1999@chapter Multiple Input Buffers 2000 2001@cindex multiple input streams 2002Some scanners (such as those which support ``include'' files) require 2003reading from several input streams. As @code{flex} scanners do a large 2004amount of buffering, one cannot control where the next input will be 2005read from by simply writing a @code{YY_INPUT()} which is sensitive to 2006the scanning context. @code{YY_INPUT()} is only called when the scanner 2007reaches the end of its buffer, which may be a long time after scanning a 2008statement such as an @code{include} statement which requires switching 2009the input source. 2010 2011To negotiate these sorts of problems, @code{flex} provides a mechanism 2012for creating and switching between multiple input buffers. An input 2013buffer is created by using: 2014 2015@cindex memory, allocating input buffers 2016@deftypefun YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size ) 2017@end deftypefun 2018 2019which takes a @code{FILE} pointer and a size and creates a buffer 2020associated with the given file and large enough to hold @code{size} 2021characters (when in doubt, use @code{YY_BUF_SIZE} for the size). It 2022returns a @code{YY_BUFFER_STATE} handle, which may then be passed to 2023other routines (see below). 2024@tindex YY_BUFFER_STATE 2025The @code{YY_BUFFER_STATE} type is a 2026pointer to an opaque @code{struct yy_buffer_state} structure, so you may 2027safely initialize @code{YY_BUFFER_STATE} variables to @code{((YY_BUFFER_STATE) 20280)} if you wish, and also refer to the opaque structure in order to 2029correctly declare input buffers in source files other than that of your 2030scanner. Note that the @code{FILE} pointer in the call to 2031@code{yy_create_buffer} is only used as the value of @file{yyin} seen by 2032@code{YY_INPUT}. If you redefine @code{YY_INPUT()} so it no longer uses 2033@file{yyin}, then you can safely pass a NULL @code{FILE} pointer to 2034@code{yy_create_buffer}. You select a particular buffer to scan from 2035using: 2036 2037@deftypefun void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer ) 2038@end deftypefun 2039 2040The above function switches the scanner's input buffer so subsequent tokens 2041will come from @code{new_buffer}. Note that @code{yy_switch_to_buffer()} may 2042be used by @code{yywrap()} to set things up for continued scanning, instead of 2043opening a new file and pointing @file{yyin} at it. If you are looking for a 2044stack of input buffers, then you want to use @code{yypush_buffer_state()} 2045instead of this function. Note also that switching input sources via either 2046@code{yy_switch_to_buffer()} or @code{yywrap()} does @emph{not} change the 2047start condition. 2048 2049@cindex memory, deleting input buffers 2050@deftypefun void yy_delete_buffer ( YY_BUFFER_STATE buffer ) 2051@end deftypefun 2052 2053is used to reclaim the storage associated with a buffer. (@code{buffer} 2054can be NULL, in which case the routine does nothing.) You can also clear 2055the current contents of a buffer using: 2056 2057@cindex pushing an input buffer 2058@cindex stack, input buffer push 2059@deftypefun void yypush_buffer_state ( YY_BUFFER_STATE buffer ) 2060@end deftypefun 2061 2062This function pushes the new buffer state onto an internal stack. The pushed 2063state becomes the new current state. The stack is maintained by flex and will 2064grow as required. This function is intended to be used instead of 2065@code{yy_switch_to_buffer}, when you want to change states, but preserve the 2066current state for later use. 2067 2068@cindex popping an input buffer 2069@cindex stack, input buffer pop 2070@deftypefun void yypop_buffer_state ( ) 2071@end deftypefun 2072 2073This function removes the current state from the top of the stack, and deletes 2074it by calling @code{yy_delete_buffer}. The next state on the stack, if any, 2075becomes the new current state. 2076 2077@cindex clearing an input buffer 2078@cindex flushing an input buffer 2079@deftypefun void yy_flush_buffer ( YY_BUFFER_STATE buffer ) 2080@end deftypefun 2081 2082This function discards the buffer's contents, 2083so the next time the scanner attempts to match a token from the 2084buffer, it will first fill the buffer anew using 2085@code{YY_INPUT()}. 2086 2087@deftypefun YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size ) 2088@end deftypefun 2089 2090is an alias for @code{yy_create_buffer()}, 2091provided for compatibility with the C++ use of @code{new} and 2092@code{delete} for creating and destroying dynamic objects. 2093 2094@cindex YY_CURRENT_BUFFER, and multiple buffers Finally, the macro 2095@code{YY_CURRENT_BUFFER} macro returns a @code{YY_BUFFER_STATE} handle to the 2096current buffer. It should not be used as an lvalue. 2097 2098@cindex EOF, example using multiple input buffers 2099Here are two examples of using these features for writing a scanner 2100which expands include files (the 2101@code{<<EOF>>} 2102feature is discussed below). 2103 2104This first example uses yypush_buffer_state and yypop_buffer_state. Flex 2105maintains the stack internally. 2106 2107@cindex handling include files with multiple input buffers 2108@example 2109@verbatim 2110 /* the "incl" state is used for picking up the name 2111 * of an include file 2112 */ 2113 %x incl 2114 %% 2115 include BEGIN(incl); 2116 2117 [a-z]+ ECHO; 2118 [^a-z\n]*\n? ECHO; 2119 2120 <incl>[ \t]* /* eat the whitespace */ 2121 <incl>[^ \t\n]+ { /* got the include file name */ 2122 yyin = fopen( yytext, "r" ); 2123 2124 if ( ! yyin ) 2125 error( ... ); 2126 2127 yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE )); 2128 2129 BEGIN(INITIAL); 2130 } 2131 2132 <<EOF>> { 2133 yypop_buffer_state(); 2134 2135 if ( !YY_CURRENT_BUFFER ) 2136 { 2137 yyterminate(); 2138 } 2139 } 2140@end verbatim 2141@end example 2142 2143The second example, below, does the same thing as the previous example did, but 2144manages its own input buffer stack manually (instead of letting flex do it). 2145 2146@cindex handling include files with multiple input buffers 2147@example 2148@verbatim 2149 /* the "incl" state is used for picking up the name 2150 * of an include file 2151 */ 2152 %x incl 2153 2154 %{ 2155 #define MAX_INCLUDE_DEPTH 10 2156 YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH]; 2157 int include_stack_ptr = 0; 2158 %} 2159 2160 %% 2161 include BEGIN(incl); 2162 2163 [a-z]+ ECHO; 2164 [^a-z\n]*\n? ECHO; 2165 2166 <incl>[ \t]* /* eat the whitespace */ 2167 <incl>[^ \t\n]+ { /* got the include file name */ 2168 if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) 2169 { 2170 fprintf( stderr, "Includes nested too deeply" ); 2171 exit( 1 ); 2172 } 2173 2174 include_stack[include_stack_ptr++] = 2175 YY_CURRENT_BUFFER; 2176 2177 yyin = fopen( yytext, "r" ); 2178 2179 if ( ! yyin ) 2180 error( ... ); 2181 2182 yy_switch_to_buffer( 2183 yy_create_buffer( yyin, YY_BUF_SIZE ) ); 2184 2185 BEGIN(INITIAL); 2186 } 2187 2188 <<EOF>> { 2189 if ( --include_stack_ptr 0 ) 2190 { 2191 yyterminate(); 2192 } 2193 2194 else 2195 { 2196 yy_delete_buffer( YY_CURRENT_BUFFER ); 2197 yy_switch_to_buffer( 2198 include_stack[include_stack_ptr] ); 2199 } 2200 } 2201@end verbatim 2202@end example 2203 2204@anchor{Scanning Strings} 2205@cindex strings, scanning strings instead of files 2206The following routines are available for setting up input buffers for 2207scanning in-memory strings instead of files. All of them create a new 2208input buffer for scanning the string, and return a corresponding 2209@code{YY_BUFFER_STATE} handle (which you should delete with 2210@code{yy_delete_buffer()} when done with it). They also switch to the 2211new buffer using @code{yy_switch_to_buffer()}, so the next call to 2212@code{yylex()} will start scanning the string. 2213 2214@deftypefun YY_BUFFER_STATE yy_scan_string ( const char *str ) 2215scans a NUL-terminated string. 2216@end deftypefun 2217 2218@deftypefun YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int len ) 2219scans @code{len} bytes (including possibly @code{NUL}s) starting at location 2220@code{bytes}. 2221@end deftypefun 2222 2223Note that both of these functions create and scan a @emph{copy} of the 2224string or bytes. (This may be desirable, since @code{yylex()} modifies 2225the contents of the buffer it is scanning.) You can avoid the copy by 2226using: 2227 2228@vindex YY_END_OF_BUFFER_CHAR 2229@deftypefun YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t size) 2230which scans in place the buffer starting at @code{base}, consisting of 2231@code{size} bytes, the last two bytes of which @emph{must} be 2232@code{YY_END_OF_BUFFER_CHAR} (ASCII NUL). These last two bytes are not 2233scanned; thus, scanning consists of @code{base[0]} through 2234@code{base[size-2]}, inclusive. 2235@end deftypefun 2236 2237If you fail to set up @code{base} in this manner (i.e., forget the final 2238two @code{YY_END_OF_BUFFER_CHAR} bytes), then @code{yy_scan_buffer()} 2239returns a NULL pointer instead of creating a new input buffer. 2240 2241@deftp {Data type} yy_size_t 2242is an integral type to which you can cast an integer expression 2243reflecting the size of the buffer. 2244@end deftp 2245 2246@node EOF, Misc Macros, Multiple Input Buffers, Top 2247@chapter End-of-File Rules 2248 2249@cindex EOF, explanation 2250The special rule @code{<<EOF>>} indicates 2251actions which are to be taken when an end-of-file is 2252encountered and @code{yywrap()} returns non-zero (i.e., indicates 2253no further files to process). The action must finish 2254by doing one of the following things: 2255 2256@itemize 2257@item 2258@findex YY_NEW_FILE (now obsolete) 2259assigning @file{yyin} to a new input file (in previous versions of 2260@code{flex}, after doing the assignment you had to call the special 2261action @code{YY_NEW_FILE}. This is no longer necessary.) 2262 2263@item 2264executing a @code{return} statement; 2265 2266@item 2267executing the special @code{yyterminate()} action. 2268 2269@item 2270or, switching to a new buffer using @code{yy_switch_to_buffer()} as 2271shown in the example above. 2272@end itemize 2273 2274<<EOF>> rules may not be used with other patterns; they may only be 2275qualified with a list of start conditions. If an unqualified <<EOF>> 2276rule is given, it applies to @emph{all} start conditions which do not 2277already have <<EOF>> actions. To specify an <<EOF>> rule for only the 2278initial start condition, use: 2279 2280@example 2281@verbatim 2282 <INITIAL><<EOF>> 2283@end verbatim 2284@end example 2285 2286These rules are useful for catching things like unclosed comments. An 2287example: 2288 2289@cindex <<EOF>>, use of 2290@example 2291@verbatim 2292 %x quote 2293 %% 2294 2295 ...other rules for dealing with quotes... 2296 2297 <quote><<EOF>> { 2298 error( "unterminated quote" ); 2299 yyterminate(); 2300 } 2301 <<EOF>> { 2302 if ( *++filelist ) 2303 yyin = fopen( *filelist, "r" ); 2304 else 2305 yyterminate(); 2306 } 2307@end verbatim 2308@end example 2309 2310@node Misc Macros, User Values, EOF, Top 2311@chapter Miscellaneous Macros 2312 2313@hkindex YY_USER_ACTION 2314The macro @code{YY_USER_ACTION} can be defined to provide an action 2315which is always executed prior to the matched rule's action. For 2316example, it could be #define'd to call a routine to convert yytext to 2317lower-case. When @code{YY_USER_ACTION} is invoked, the variable 2318@code{yy_act} gives the number of the matched rule (rules are numbered 2319starting with 1). Suppose you want to profile how often each of your 2320rules is matched. The following would do the trick: 2321 2322@cindex YY_USER_ACTION to track each time a rule is matched 2323@example 2324@verbatim 2325 #define YY_USER_ACTION ++ctr[yy_act] 2326@end verbatim 2327@end example 2328 2329@vindex YY_NUM_RULES 2330where @code{ctr} is an array to hold the counts for the different rules. 2331Note that the macro @code{YY_NUM_RULES} gives the total number of rules 2332(including the default rule), even if you use @samp{-s)}, so a correct 2333declaration for @code{ctr} is: 2334 2335@example 2336@verbatim 2337 int ctr[YY_NUM_RULES]; 2338@end verbatim 2339@end example 2340 2341@hkindex YY_USER_INIT 2342The macro @code{YY_USER_INIT} may be defined to provide an action which 2343is always executed before the first scan (and before the scanner's 2344internal initializations are done). For example, it could be used to 2345call a routine to read in a data table or open a logging file. 2346 2347@findex yy_set_interactive 2348The macro @code{yy_set_interactive(is_interactive)} can be used to 2349control whether the current buffer is considered @dfn{interactive}. An 2350interactive buffer is processed more slowly, but must be used when the 2351scanner's input source is indeed interactive to avoid problems due to 2352waiting to fill buffers (see the discussion of the @samp{-I} flag in 2353@ref{Scanner Options}). A non-zero value in the macro invocation marks 2354the buffer as interactive, a zero value as non-interactive. Note that 2355use of this macro overrides @code{%option always-interactive} or 2356@code{%option never-interactive} (@pxref{Scanner Options}). 2357@code{yy_set_interactive()} must be invoked prior to beginning to scan 2358the buffer that is (or is not) to be considered interactive. 2359 2360@cindex BOL, setting it 2361@findex yy_set_bol 2362The macro @code{yy_set_bol(at_bol)} can be used to control whether the 2363current buffer's scanning context for the next token match is done as 2364though at the beginning of a line. A non-zero macro argument makes 2365rules anchored with @samp{^} active, while a zero argument makes 2366@samp{^} rules inactive. 2367 2368@cindex BOL, checking the BOL flag 2369@findex YY_AT_BOL 2370The macro @code{YY_AT_BOL()} returns true if the next token scanned from 2371the current buffer will have @samp{^} rules active, false otherwise. 2372 2373@cindex actions, redefining YY_BREAK 2374@hkindex YY_BREAK 2375In the generated scanner, the actions are all gathered in one large 2376switch statement and separated using @code{YY_BREAK}, which may be 2377redefined. By default, it is simply a @code{break}, to separate each 2378rule's action from the following rule's. Redefining @code{YY_BREAK} 2379allows, for example, C++ users to #define YY_BREAK to do nothing (while 2380being very careful that every rule ends with a @code{break} or a 2381@code{return}!) to avoid suffering from unreachable statement warnings 2382where because a rule's action ends with @code{return}, the 2383@code{YY_BREAK} is inaccessible. 2384 2385@node User Values, Yacc, Misc Macros, Top 2386@chapter Values Available To the User 2387 2388This chapter summarizes the various values available to the user in the 2389rule actions. 2390 2391@table @code 2392@vindex yytext 2393@item char *yytext 2394holds the text of the current token. It may be modified but not 2395lengthened (you cannot append characters to the end). 2396 2397@cindex yytext, default array size 2398@cindex array, default size for yytext 2399@vindex YYLMAX 2400If the special directive @code{%array} appears in the first section of 2401the scanner description, then @code{yytext} is instead declared 2402@code{char yytext[YYLMAX]}, where @code{YYLMAX} is a macro definition 2403that you can redefine in the first section if you don't like the default 2404value (generally 8KB). Using @code{%array} results in somewhat slower 2405scanners, but the value of @code{yytext} becomes immune to calls to 2406@code{unput()}, which potentially destroy its value when @code{yytext} is 2407a character pointer. The opposite of @code{%array} is @code{%pointer}, 2408which is the default. 2409 2410@cindex C++ and %array 2411You cannot use @code{%array} when generating C++ scanner classes (the 2412@samp{-+} flag). 2413 2414@vindex yyleng 2415@item int yyleng 2416holds the length of the current token. 2417 2418@vindex yyin 2419@item FILE *yyin 2420is the file which by default @code{flex} reads from. It may be 2421redefined but doing so only makes sense before scanning begins or after 2422an EOF has been encountered. Changing it in the midst of scanning will 2423have unexpected results since @code{flex} buffers its input; use 2424@code{yyrestart()} instead. Once scanning terminates because an 2425end-of-file has been seen, you can assign @file{yyin} at the new input 2426file and then call the scanner again to continue scanning. 2427 2428@findex yyrestart 2429@item void yyrestart( FILE *new_file ) 2430may be called to point @file{yyin} at the new input file. The 2431switch-over to the new file is immediate (any previously buffered-up 2432input is lost). Note that calling @code{yyrestart()} with @file{yyin} 2433as an argument thus throws away the current input buffer and continues 2434scanning the same input file. 2435 2436@vindex yyout 2437@item FILE *yyout 2438is the file to which @code{ECHO} actions are done. It can be reassigned 2439by the user. 2440 2441@vindex YY_CURRENT_BUFFER 2442@item YY_CURRENT_BUFFER 2443returns a @code{YY_BUFFER_STATE} handle to the current buffer. 2444 2445@vindex YY_START 2446@item YY_START 2447returns an integer value corresponding to the current start condition. 2448You can subsequently use this value with @code{BEGIN} to return to that 2449start condition. 2450@end table 2451 2452@node Yacc, Scanner Options, User Values, Top 2453@chapter Interfacing with Yacc 2454 2455@cindex yacc, interface 2456 2457@vindex yylval, with yacc 2458One of the main uses of @code{flex} is as a companion to the @code{yacc} 2459parser-generator. @code{yacc} parsers expect to call a routine named 2460@code{yylex()} to find the next input token. The routine is supposed to 2461return the type of the next token as well as putting any associated 2462value in the global @code{yylval}. To use @code{flex} with @code{yacc}, 2463one specifies the @samp{-d} option to @code{yacc} to instruct it to 2464generate the file @file{y.tab.h} containing definitions of all the 2465@code{%tokens} appearing in the @code{yacc} input. This file is then 2466included in the @code{flex} scanner. For example, if one of the tokens 2467is @code{TOK_NUMBER}, part of the scanner might look like: 2468 2469@cindex yacc interface 2470@example 2471@verbatim 2472 %{ 2473 #include "y.tab.h" 2474 %} 2475 2476 %% 2477 2478 [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER; 2479@end verbatim 2480@end example 2481 2482@node Scanner Options, Performance, Yacc, Top 2483@chapter Scanner Options 2484 2485@cindex command-line options 2486@cindex options, command-line 2487@cindex arguments, command-line 2488 2489The various @code{flex} options are categorized by function in the following 2490menu. If you want to lookup a particular option by name, @xref{Index of Scanner Options}. 2491 2492@menu 2493* Options for Specifying Filenames:: 2494* Options Affecting Scanner Behavior:: 2495* Code-Level And API Options:: 2496* Options for Scanner Speed and Size:: 2497* Debugging Options:: 2498* Miscellaneous Options:: 2499@end menu 2500 2501Even though there are many scanner options, a typical scanner might only 2502specify the following options: 2503 2504@example 2505@verbatim 2506%option 8bit reentrant bison-bridge 2507%option warn nodefault 2508%option yylineno 2509%option outfile="scanner.c" header-file="scanner.h" 2510@end verbatim 2511@end example 2512 2513The first line specifies the general type of scanner we want. The second line 2514specifies that we are being careful. The third line asks flex to track line 2515numbers. The last line tells flex what to name the files. (The options can be 2516specified in any order. We just divided them.) 2517 2518@code{flex} also provides a mechanism for controlling options within the 2519scanner specification itself, rather than from the flex command-line. 2520This is done by including @code{%option} directives in the first section 2521of the scanner specification. You can specify multiple options with a 2522single @code{%option} directive, and multiple directives in the first 2523section of your flex input file. 2524 2525Most options are given simply as names, optionally preceded by the 2526word @samp{no} (with no intervening whitespace) to negate their meaning. 2527The names are the same as their long-option equivalents (but without the 2528leading @samp{--} ). 2529 2530@code{flex} scans your rule actions to determine whether you use the 2531@code{REJECT} or @code{yymore()} features. The @code{REJECT} and 2532@code{yymore} options are available to override its decision as to 2533whether you use the options, either by setting them (e.g., @code{%option 2534reject)} to indicate the feature is indeed used, or unsetting them to 2535indicate it actually is not used (e.g., @code{%option noyymore)}. 2536 2537 2538A number of options are available for lint purists who want to suppress 2539the appearance of unneeded routines in the generated scanner. Each of 2540the following, if unset (e.g., @code{%option nounput}), results in the 2541corresponding routine not appearing in the generated scanner: 2542 2543@example 2544@verbatim 2545 input, unput 2546 yy_push_state, yy_pop_state, yy_top_state 2547 yy_scan_buffer, yy_scan_bytes, yy_scan_string 2548 2549 yyget_extra, yyset_extra, yyget_leng, yyget_text, 2550 yyget_lineno, yyset_lineno, yyget_in, yyset_in, 2551 yyget_out, yyset_out, yyget_lval, yyset_lval, 2552 yyget_lloc, yyset_lloc, yyget_debug, yyset_debug 2553@end verbatim 2554@end example 2555 2556(though @code{yy_push_state()} and friends won't appear anyway unless 2557you use @code{%option stack)}. 2558 2559@node Options for Specifying Filenames, Options Affecting Scanner Behavior, Scanner Options, Scanner Options 2560@section Options for Specifying Filenames 2561 2562@table @samp 2563 2564@anchor{option-header} 2565@opindex ---header-file 2566@opindex header-file 2567@item --header-file=FILE, @code{%option header-file="FILE"} 2568instructs flex to write a C header to @file{FILE}. This file contains 2569function prototypes, extern variables, and types used by the scanner. 2570Only the external API is exported by the header file. Many macros that 2571are usable from within scanner actions are not exported to the header 2572file. This is due to namespace problems and the goal of a clean 2573external API. 2574 2575While in the header, the macro @code{yyIN_HEADER} is defined, where @samp{yy} 2576is substituted with the appropriate prefix. 2577 2578The @samp{--header-file} option is not compatible with the @samp{--c++} option, 2579since the C++ scanner provides its own header in @file{yyFlexLexer.h}. 2580 2581 2582 2583@anchor{option-outfile} 2584@opindex -o 2585@opindex ---outfile 2586@opindex outfile 2587@item -oFILE, --outfile=FILE, @code{%option outfile="FILE"} 2588directs flex to write the scanner to the file @file{FILE} instead of 2589@file{lex.yy.c}. If you combine @samp{--outfile} with the @samp{--stdout} option, 2590then the scanner is written to @file{stdout} but its @code{#line} 2591directives (see the @samp{-l} option above) refer to the file 2592@file{FILE}. 2593 2594 2595 2596@anchor{option-stdout} 2597@opindex -t 2598@opindex ---stdout 2599@opindex stdout 2600@item -t, --stdout, @code{%option stdout} 2601instructs @code{flex} to write the scanner it generates to standard 2602output instead of @file{lex.yy.c}. 2603 2604 2605 2606@opindex ---skel 2607@item -SFILE, --skel=FILE 2608overrides the default skeleton file from which 2609@code{flex} 2610constructs its scanners. You'll never need this option unless you are doing 2611@code{flex} 2612maintenance or development. 2613 2614@opindex ---tables-file 2615@opindex tables-file 2616@item --tables-file=FILE 2617Write serialized scanner dfa tables to FILE. The generated scanner will not 2618contain the tables, and requires them to be loaded at runtime. 2619@xref{serialization}. 2620 2621@opindex ---tables-verify 2622@opindex tables-verify 2623@item --tables-verify 2624This option is for flex development. We document it here in case you stumble 2625upon it by accident or in case you suspect some inconsistency in the serialized 2626tables. Flex will serialize the scanner dfa tables but will also generate the 2627in-code tables as it normally does. At runtime, the scanner will verify that 2628the serialized tables match the in-code tables, instead of loading them. 2629 2630@end table 2631 2632@node Options Affecting Scanner Behavior, Code-Level And API Options, Options for Specifying Filenames, Scanner Options 2633@section Options Affecting Scanner Behavior 2634 2635@table @samp 2636@anchor{option-case-insensitive} 2637@opindex -i 2638@opindex ---case-insensitive 2639@opindex case-insensitive 2640@item -i, --case-insensitive, @code{%option case-insensitive} 2641instructs @code{flex} to generate a @dfn{case-insensitive} scanner. The 2642case of letters given in the @code{flex} input patterns will be ignored, 2643and tokens in the input will be matched regardless of case. The matched 2644text given in @code{yytext} will have the preserved case (i.e., it will 2645not be folded). For tricky behavior, see @ref{case and character ranges}. 2646 2647 2648 2649@anchor{option-lex-compat} 2650@opindex -l 2651@opindex ---lex-compat 2652@opindex lex-compat 2653@item -l, --lex-compat, @code{%option lex-compat} 2654turns on maximum compatibility with the original AT&T @code{lex} 2655implementation. Note that this does not mean @emph{full} compatibility. 2656Use of this option costs a considerable amount of performance, and it 2657cannot be used with the @samp{--c++}, @samp{--full}, @samp{--fast}, @samp{-Cf}, or 2658@samp{-CF} options. For details on the compatibilities it provides, see 2659@ref{Lex and Posix}. This option also results in the name 2660@code{YY_FLEX_LEX_COMPAT} being @code{#define}'d in the generated scanner. 2661 2662 2663 2664@anchor{option-batch} 2665@opindex -B 2666@opindex ---batch 2667@opindex batch 2668@item -B, --batch, @code{%option batch} 2669instructs @code{flex} to generate a @dfn{batch} scanner, the opposite of 2670@emph{interactive} scanners generated by @samp{--interactive} (see below). In 2671general, you use @samp{-B} when you are @emph{certain} that your scanner 2672will never be used interactively, and you want to squeeze a 2673@emph{little} more performance out of it. If your goal is instead to 2674squeeze out a @emph{lot} more performance, you should be using the 2675@samp{-Cf} or @samp{-CF} options, which turn on @samp{--batch} automatically 2676anyway. 2677 2678 2679 2680@anchor{option-interactive} 2681@opindex -I 2682@opindex ---interactive 2683@opindex interactive 2684@item -I, --interactive, @code{%option interactive} 2685instructs @code{flex} to generate an @i{interactive} scanner. An 2686interactive scanner is one that only looks ahead to decide what token 2687has been matched if it absolutely must. It turns out that always 2688looking one extra character ahead, even if the scanner has already seen 2689enough text to disambiguate the current token, is a bit faster than only 2690looking ahead when necessary. But scanners that always look ahead give 2691dreadful interactive performance; for example, when a user types a 2692newline, it is not recognized as a newline token until they enter 2693@emph{another} token, which often means typing in another whole line. 2694 2695@code{flex} scanners default to @code{interactive} unless you use the 2696@samp{-Cf} or @samp{-CF} table-compression options 2697(@pxref{Performance}). That's because if you're looking for 2698high-performance you should be using one of these options, so if you 2699didn't, @code{flex} assumes you'd rather trade off a bit of run-time 2700performance for intuitive interactive behavior. Note also that you 2701@emph{cannot} use @samp{--interactive} in conjunction with @samp{-Cf} or 2702@samp{-CF}. Thus, this option is not really needed; it is on by default 2703for all those cases in which it is allowed. 2704 2705You can force a scanner to 2706@emph{not} 2707be interactive by using 2708@samp{--batch} 2709 2710 2711 2712@anchor{option-7bit} 2713@opindex -7 2714@opindex ---7bit 2715@opindex 7bit 2716@item -7, --7bit, @code{%option 7bit} 2717instructs @code{flex} to generate a 7-bit scanner, i.e., one which can 2718only recognize 7-bit characters in its input. The advantage of using 2719@samp{--7bit} is that the scanner's tables can be up to half the size of 2720those generated using the @samp{--8bit}. The disadvantage is that such 2721scanners often hang or crash if their input contains an 8-bit character. 2722 2723Note, however, that unless you generate your scanner using the 2724@samp{-Cf} or @samp{-CF} table compression options, use of @samp{--7bit} 2725will save only a small amount of table space, and make your scanner 2726considerably less portable. @code{Flex}'s default behavior is to 2727generate an 8-bit scanner unless you use the @samp{-Cf} or @samp{-CF}, 2728in which case @code{flex} defaults to generating 7-bit scanners unless 2729your site was always configured to generate 8-bit scanners (as will 2730often be the case with non-USA sites). You can tell whether flex 2731generated a 7-bit or an 8-bit scanner by inspecting the flag summary in 2732the @samp{--verbose} output as described above. 2733 2734Note that if you use @samp{-Cfe} or @samp{-CFe} @code{flex} still 2735defaults to generating an 8-bit scanner, since usually with these 2736compression options full 8-bit tables are not much more expensive than 27377-bit tables. 2738 2739 2740 2741@anchor{option-8bit} 2742@opindex -8 2743@opindex ---8bit 2744@opindex 8bit 2745@item -8, --8bit, @code{%option 8bit} 2746instructs @code{flex} to generate an 8-bit scanner, i.e., one which can 2747recognize 8-bit characters. This flag is only needed for scanners 2748generated using @samp{-Cf} or @samp{-CF}, as otherwise flex defaults to 2749generating an 8-bit scanner anyway. 2750 2751See the discussion of 2752@samp{--7bit} 2753above for @code{flex}'s default behavior and the tradeoffs between 7-bit 2754and 8-bit scanners. 2755 2756 2757 2758@anchor{option-default} 2759@opindex ---default 2760@opindex default 2761@item --default, @code{%option default} 2762generate the default rule. 2763 2764 2765 2766@anchor{option-always-interactive} 2767@opindex ---always-interactive 2768@opindex always-interactive 2769@item --always-interactive, @code{%option always-interactive} 2770instructs flex to generate a scanner which always considers its input 2771@emph{interactive}. Normally, on each new input file the scanner calls 2772@code{isatty()} in an attempt to determine whether the scanner's input 2773source is interactive and thus should be read a character at a time. 2774When this option is used, however, then no such call is made. 2775 2776 2777 2778@opindex ---never-interactive 2779@item --never-interactive, @code{--never-interactive} 2780instructs flex to generate a scanner which never considers its input 2781interactive. This is the opposite of @code{always-interactive}. 2782 2783 2784@anchor{option-posix} 2785@opindex -X 2786@opindex ---posix 2787@opindex posix 2788@item -X, --posix, @code{%option posix} 2789turns on maximum compatibility with the POSIX 1003.2-1992 definition of 2790@code{lex}. Since @code{flex} was originally designed to implement the 2791POSIX definition of @code{lex} this generally involves very few changes 2792in behavior. At the current writing the known differences between 2793@code{flex} and the POSIX standard are: 2794 2795@itemize 2796@item 2797In POSIX and AT&T @code{lex}, the repeat operator, @samp{@{@}}, has lower 2798precedence than concatenation (thus @samp{ab@{3@}} yields @samp{ababab}). 2799Most POSIX utilities use an Extended Regular Expression (ERE) precedence 2800that has the precedence of the repeat operator higher than concatenation 2801(which causes @samp{ab@{3@}} to yield @samp{abbb}). By default, @code{flex} 2802places the precedence of the repeat operator higher than concatenation 2803which matches the ERE processing of other POSIX utilities. When either 2804@samp{--posix} or @samp{-l} are specified, @code{flex} will use the 2805traditional AT&T and POSIX-compliant precedence for the repeat operator 2806where concatenation has higher precedence than the repeat operator. 2807@end itemize 2808 2809 2810@anchor{option-stack} 2811@opindex ---stack 2812@opindex stack 2813@item --stack, @code{%option stack} 2814enables the use of 2815start condition stacks (@pxref{Start Conditions}). 2816 2817 2818 2819@anchor{option-stdinit} 2820@opindex ---stdinit 2821@opindex stdinit 2822@item --stdinit, @code{%option stdinit} 2823if set (i.e., @b{%option stdinit)} initializes @code{yyin} and 2824@code{yyout} to @file{stdin} and @file{stdout}, instead of the default of 2825@file{NULL}. Some existing @code{lex} programs depend on this behavior, 2826even though it is not compliant with ANSI C, which does not require 2827@file{stdin} and @file{stdout} to be compile-time constant. In a 2828reentrant scanner, however, this is not a problem since initialization 2829is performed in @code{yylex_init} at runtime. 2830 2831 2832 2833@anchor{option-yylineno} 2834@opindex ---yylineno 2835@opindex yylineno 2836@item --yylineno, @code{%option yylineno} 2837directs @code{flex} to generate a scanner 2838that maintains the number of the current line read from its input in the 2839global variable @code{yylineno}. This option is implied by @code{%option 2840lex-compat}. In a reentrant C scanner, the macro @code{yylineno} is 2841accessible regardless of the value of @code{%option yylineno}, however, its 2842value is not modified by @code{flex} unless @code{%option yylineno} is enabled. 2843 2844 2845 2846@anchor{option-yywrap} 2847@opindex ---yywrap 2848@opindex yywrap 2849@item --yywrap, @code{%option yywrap} 2850if unset (i.e., @code{--noyywrap)}, makes the scanner not call 2851@code{yywrap()} upon an end-of-file, but simply assume that there are no 2852more files to scan (until the user points @file{yyin} at a new file and 2853calls @code{yylex()} again). 2854 2855@end table 2856 2857@node Code-Level And API Options, Options for Scanner Speed and Size, Options Affecting Scanner Behavior, Scanner Options 2858@section Code-Level And API Options 2859 2860@table @samp 2861 2862@anchor{option-ansi-definitions} 2863@opindex ---option-ansi-definitions 2864@opindex ansi-definitions 2865@item --ansi-definitions, @code{%option ansi-definitions} 2866instruct flex to generate ANSI C99 definitions for functions. 2867This option is enabled by default. 2868If @code{%option noansi-definitions} is specified, then the obsolete style 2869is generated. 2870 2871@anchor{option-ansi-prototypes} 2872@opindex ---option-ansi-prototypes 2873@opindex ansi-prototypes 2874@item --ansi-prototypes, @code{%option ansi-prototypes} 2875instructs flex to generate ANSI C99 prototypes for functions. 2876This option is enabled by default. 2877If @code{noansi-prototypes} is specified, then 2878prototypes will have empty parameter lists. 2879 2880@anchor{option-bison-bridge} 2881@opindex ---bison-bridge 2882@opindex bison-bridge 2883@item --bison-bridge, @code{%option bison-bridge} 2884instructs flex to generate a C scanner that is 2885meant to be called by a 2886@code{GNU bison} 2887parser. The scanner has minor API changes for 2888@code{bison} 2889compatibility. In particular, the declaration of 2890@code{yylex} 2891is modified to take an additional parameter, 2892@code{yylval}. 2893@xref{Bison Bridge}. 2894 2895@anchor{option-bison-locations} 2896@opindex ---bison-locations 2897@opindex bison-locations 2898@item --bison-locations, @code{%option bison-locations} 2899instruct flex that 2900@code{GNU bison} @code{%locations} are being used. 2901This means @code{yylex} will be passed 2902an additional parameter, @code{yylloc}. This option 2903implies @code{%option bison-bridge}. 2904@xref{Bison Bridge}. 2905 2906@anchor{option-noline} 2907@opindex -L 2908@opindex ---noline 2909@opindex noline 2910@item -L, --noline, @code{%option noline} 2911instructs 2912@code{flex} 2913not to generate 2914@code{#line} 2915directives. Without this option, 2916@code{flex} 2917peppers the generated scanner 2918with @code{#line} directives so error messages in the actions will be correctly 2919located with respect to either the original 2920@code{flex} 2921input file (if the errors are due to code in the input file), or 2922@file{lex.yy.c} 2923(if the errors are 2924@code{flex}'s 2925fault -- you should report these sorts of errors to the email address 2926given in @ref{Reporting Bugs}). 2927 2928 2929 2930@anchor{option-reentrant} 2931@opindex -R 2932@opindex ---reentrant 2933@opindex reentrant 2934@item -R, --reentrant, @code{%option reentrant} 2935instructs flex to generate a reentrant C scanner. The generated scanner 2936may safely be used in a multi-threaded environment. The API for a 2937reentrant scanner is different than for a non-reentrant scanner 2938@pxref{Reentrant}). Because of the API difference between 2939reentrant and non-reentrant @code{flex} scanners, non-reentrant flex 2940code must be modified before it is suitable for use with this option. 2941This option is not compatible with the @samp{--c++} option. 2942 2943The option @samp{--reentrant} does not affect the performance of 2944the scanner. 2945 2946 2947 2948@anchor{option-c++} 2949@opindex -+ 2950@opindex ---c++ 2951@opindex c++ 2952@item -+, --c++, @code{%option c++} 2953specifies that you want flex to generate a C++ 2954scanner class. @xref{Cxx}, for 2955details. 2956 2957 2958 2959@anchor{option-array} 2960@opindex ---array 2961@opindex array 2962@item --array, @code{%option array} 2963specifies that you want yytext to be an array instead of a char* 2964 2965 2966 2967@anchor{option-pointer} 2968@opindex ---pointer 2969@opindex pointer 2970@item --pointer, @code{%option pointer} 2971specify that @code{yytext} should be a @code{char *}, not an array. 2972This default is @code{char *}. 2973 2974 2975 2976@anchor{option-prefix} 2977@opindex -P 2978@opindex ---prefix 2979@opindex prefix 2980@item -PPREFIX, --prefix=PREFIX, @code{%option prefix="PREFIX"} 2981changes the default @samp{yy} prefix used by @code{flex} for all 2982globally-visible variable and function names to instead be 2983@samp{PREFIX}. For example, @samp{--prefix=foo} changes the name of 2984@code{yytext} to @code{footext}. It also changes the name of the default 2985output file from @file{lex.yy.c} to @file{lex.foo.c}. Here is a partial 2986list of the names affected: 2987 2988@example 2989@verbatim 2990 yy_create_buffer 2991 yy_delete_buffer 2992 yy_flex_debug 2993 yy_init_buffer 2994 yy_flush_buffer 2995 yy_load_buffer_state 2996 yy_switch_to_buffer 2997 yyin 2998 yyleng 2999 yylex 3000 yylineno 3001 yyout 3002 yyrestart 3003 yytext 3004 yywrap 3005 yyalloc 3006 yyrealloc 3007 yyfree 3008@end verbatim 3009@end example 3010 3011(If you are using a C++ scanner, then only @code{yywrap} and 3012@code{yyFlexLexer} are affected.) Within your scanner itself, you can 3013still refer to the global variables and functions using either version 3014of their name; but externally, they have the modified name. 3015 3016This option lets you easily link together multiple 3017@code{flex} 3018programs into the same executable. Note, though, that using this 3019option also renames 3020@code{yywrap()}, 3021so you now 3022@emph{must} 3023either 3024provide your own (appropriately-named) version of the routine for your 3025scanner, or use 3026@code{%option noyywrap}, 3027as linking with 3028@samp{-lfl} 3029no longer provides one for you by default. 3030 3031 3032 3033@anchor{option-main} 3034@opindex ---main 3035@opindex main 3036@item --main, @code{%option main} 3037 directs flex to provide a default @code{main()} program for the 3038scanner, which simply calls @code{yylex()}. This option implies 3039@code{noyywrap} (see below). 3040 3041 3042 3043@anchor{option-nounistd} 3044@opindex ---nounistd 3045@opindex nounistd 3046@item --nounistd, @code{%option nounistd} 3047suppresses inclusion of the non-ANSI header file @file{unistd.h}. This option 3048is meant to target environments in which @file{unistd.h} does not exist. Be aware 3049that certain options may cause flex to generate code that relies on functions 3050normally found in @file{unistd.h}, (e.g. @code{isatty()}, @code{read()}.) 3051If you wish to use these functions, you will have to inform your compiler where 3052to find them. 3053@xref{option-always-interactive}. @xref{option-read}. 3054 3055 3056 3057@anchor{option-yyclass} 3058@opindex ---yyclass 3059@opindex yyclass 3060@item --yyclass=NAME, @code{%option yyclass="NAME"} 3061only applies when generating a C++ scanner (the @samp{--c++} option). It 3062informs @code{flex} that you have derived @code{NAME} as a subclass of 3063@code{yyFlexLexer}, so @code{flex} will place your actions in the member 3064function @code{foo::yylex()} instead of @code{yyFlexLexer::yylex()}. It 3065also generates a @code{yyFlexLexer::yylex()} member function that emits 3066a run-time error (by invoking @code{yyFlexLexer::LexerError())} if 3067called. @xref{Cxx}. 3068 3069@end table 3070 3071@node Options for Scanner Speed and Size, Debugging Options, Code-Level And API Options, Scanner Options 3072@section Options for Scanner Speed and Size 3073 3074@table @samp 3075 3076@item -C[aefFmr] 3077controls the degree of table compression and, more generally, trade-offs 3078between small scanners and fast scanners. 3079 3080@table @samp 3081@opindex -C 3082@item -C 3083A lone @samp{-C} specifies that the scanner tables should be compressed 3084but neither equivalence classes nor meta-equivalence classes should be 3085used. 3086 3087@anchor{option-align} 3088@opindex -Ca 3089@opindex ---align 3090@opindex align 3091@item -Ca, --align, @code{%option align} 3092(``align'') instructs flex to trade off larger tables in the 3093generated scanner for faster performance because the elements of 3094the tables are better aligned for memory access and computation. On some 3095RISC architectures, fetching and manipulating longwords is more efficient 3096than with smaller-sized units such as shortwords. This option can 3097quadruple the size of the tables used by your scanner. 3098 3099@anchor{option-ecs} 3100@opindex -Ce 3101@opindex ---ecs 3102@opindex ecs 3103@item -Ce, --ecs, @code{%option ecs} 3104directs @code{flex} to construct @dfn{equivalence classes}, i.e., sets 3105of characters which have identical lexical properties (for example, if 3106the only appearance of digits in the @code{flex} input is in the 3107character class ``[0-9]'' then the digits '0', '1', ..., '9' will all be 3108put in the same equivalence class). Equivalence classes usually give 3109dramatic reductions in the final table/object file sizes (typically a 3110factor of 2-5) and are pretty cheap performance-wise (one array look-up 3111per character scanned). 3112 3113@opindex -Cf 3114@item -Cf 3115specifies that the @dfn{full} scanner tables should be generated - 3116@code{flex} should not compress the tables by taking advantages of 3117similar transition functions for different states. 3118 3119@opindex -CF 3120@item -CF 3121specifies that the alternate fast scanner representation (described 3122above under the @samp{--fast} flag) should be used. This option cannot be 3123used with @samp{--c++}. 3124 3125@anchor{option-meta-ecs} 3126@opindex -Cm 3127@opindex ---meta-ecs 3128@opindex meta-ecs 3129@item -Cm, --meta-ecs, @code{%option meta-ecs} 3130directs 3131@code{flex} 3132to construct 3133@dfn{meta-equivalence classes}, 3134which are sets of equivalence classes (or characters, if equivalence 3135classes are not being used) that are commonly used together. Meta-equivalence 3136classes are often a big win when using compressed tables, but they 3137have a moderate performance impact (one or two @code{if} tests and one 3138array look-up per character scanned). 3139 3140@anchor{option-read} 3141@opindex -Cr 3142@opindex ---read 3143@opindex read 3144@item -Cr, --read, @code{%option read} 3145causes the generated scanner to @emph{bypass} use of the standard I/O 3146library (@code{stdio}) for input. Instead of calling @code{fread()} or 3147@code{getc()}, the scanner will use the @code{read()} system call, 3148resulting in a performance gain which varies from system to system, but 3149in general is probably negligible unless you are also using @samp{-Cf} 3150or @samp{-CF}. Using @samp{-Cr} can cause strange behavior if, for 3151example, you read from @file{yyin} using @code{stdio} prior to calling 3152the scanner (because the scanner will miss whatever text your previous 3153reads left in the @code{stdio} input buffer). @samp{-Cr} has no effect 3154if you define @code{YY_INPUT()} (@pxref{Generated Scanner}). 3155@end table 3156 3157The options @samp{-Cf} or @samp{-CF} and @samp{-Cm} do not make sense 3158together - there is no opportunity for meta-equivalence classes if the 3159table is not being compressed. Otherwise the options may be freely 3160mixed, and are cumulative. 3161 3162The default setting is @samp{-Cem}, which specifies that @code{flex} 3163should generate equivalence classes and meta-equivalence classes. This 3164setting provides the highest degree of table compression. You can trade 3165off faster-executing scanners at the cost of larger tables with the 3166following generally being true: 3167 3168@example 3169@verbatim 3170 slowest & smallest 3171 -Cem 3172 -Cm 3173 -Ce 3174 -C 3175 -C{f,F}e 3176 -C{f,F} 3177 -C{f,F}a 3178 fastest & largest 3179@end verbatim 3180@end example 3181 3182Note that scanners with the smallest tables are usually generated and 3183compiled the quickest, so during development you will usually want to 3184use the default, maximal compression. 3185 3186@samp{-Cfe} is often a good compromise between speed and size for 3187production scanners. 3188 3189@anchor{option-full} 3190@opindex -f 3191@opindex ---full 3192@opindex full 3193@item -f, --full, @code{%option full} 3194specifies 3195@dfn{fast scanner}. 3196No table compression is done and @code{stdio} is bypassed. 3197The result is large but fast. This option is equivalent to 3198@samp{--Cfr} 3199 3200 3201@anchor{option-fast} 3202@opindex -F 3203@opindex ---fast 3204@opindex fast 3205@item -F, --fast, @code{%option fast} 3206specifies that the @emph{fast} scanner table representation should be 3207used (and @code{stdio} bypassed). This representation is about as fast 3208as the full table representation @samp{--full}, and for some sets of 3209patterns will be considerably smaller (and for others, larger). In 3210general, if the pattern set contains both @emph{keywords} and a 3211catch-all, @emph{identifier} rule, such as in the set: 3212 3213@example 3214@verbatim 3215 "case" return TOK_CASE; 3216 "switch" return TOK_SWITCH; 3217 ... 3218 "default" return TOK_DEFAULT; 3219 [a-z]+ return TOK_ID; 3220@end verbatim 3221@end example 3222 3223then you're better off using the full table representation. If only 3224the @emph{identifier} rule is present and you then use a hash table or some such 3225to detect the keywords, you're better off using 3226@samp{--fast}. 3227 3228This option is equivalent to @samp{-CFr}. It cannot be used 3229with @samp{--c++}. 3230 3231@end table 3232 3233@node Debugging Options, Miscellaneous Options, Options for Scanner Speed and Size, Scanner Options 3234@section Debugging Options 3235 3236@table @samp 3237 3238@anchor{option-backup} 3239@opindex -b 3240@opindex ---backup 3241@opindex backup 3242@item -b, --backup, @code{%option backup} 3243Generate backing-up information to @file{lex.backup}. This is a list of 3244scanner states which require backing up and the input characters on 3245which they do so. By adding rules one can remove backing-up states. If 3246@emph{all} backing-up states are eliminated and @samp{-Cf} or @code{-CF} 3247is used, the generated scanner will run faster (see the @samp{--perf-report} flag). 3248Only users who wish to squeeze every last cycle out of their scanners 3249need worry about this option. (@pxref{Performance}). 3250 3251 3252 3253@anchor{option-debug} 3254@opindex -d 3255@opindex ---debug 3256@opindex debug 3257@item -d, --debug, @code{%option debug} 3258makes the generated scanner run in @dfn{debug} mode. Whenever a pattern 3259is recognized and the global variable @code{yy_flex_debug} is non-zero 3260(which is the default), the scanner will write to @file{stderr} a line 3261of the form: 3262 3263@example 3264@verbatim 3265 -accepting rule at line 53 ("the matched text") 3266@end verbatim 3267@end example 3268 3269The line number refers to the location of the rule in the file defining 3270the scanner (i.e., the file that was fed to flex). Messages are also 3271generated when the scanner backs up, accepts the default rule, reaches 3272the end of its input buffer (or encounters a NUL; at this point, the two 3273look the same as far as the scanner's concerned), or reaches an 3274end-of-file. 3275 3276 3277 3278@anchor{option-perf-report} 3279@opindex -p 3280@opindex ---perf-report 3281@opindex perf-report 3282@item -p, --perf-report, @code{%option perf-report} 3283generates a performance report to @file{stderr}. The report consists of 3284comments regarding features of the @code{flex} input file which will 3285cause a serious loss of performance in the resulting scanner. If you 3286give the flag twice, you will also get comments regarding features that 3287lead to minor performance losses. 3288 3289Note that the use of @code{REJECT}, and 3290variable trailing context (@pxref{Limitations}) entails a substantial 3291performance penalty; use of @code{yymore()}, the @samp{^} operator, and 3292the @samp{--interactive} flag entail minor performance penalties. 3293 3294 3295 3296@anchor{option-nodefault} 3297@opindex -s 3298@opindex ---nodefault 3299@opindex nodefault 3300@item -s, --nodefault, @code{%option nodefault} 3301causes the @emph{default rule} (that unmatched scanner input is echoed 3302to @file{stdout)} to be suppressed. If the scanner encounters input 3303that does not match any of its rules, it aborts with an error. This 3304option is useful for finding holes in a scanner's rule set. 3305 3306 3307 3308@anchor{option-trace} 3309@opindex -T 3310@opindex ---trace 3311@opindex trace 3312@item -T, --trace, @code{%option trace} 3313makes @code{flex} run in @dfn{trace} mode. It will generate a lot of 3314messages to @file{stderr} concerning the form of the input and the 3315resultant non-deterministic and deterministic finite automata. This 3316option is mostly for use in maintaining @code{flex}. 3317 3318 3319 3320@anchor{option-nowarn} 3321@opindex -w 3322@opindex ---nowarn 3323@opindex nowarn 3324@item -w, --nowarn, @code{%option nowarn} 3325suppresses warning messages. 3326 3327 3328 3329@anchor{option-verbose} 3330@opindex -v 3331@opindex ---verbose 3332@opindex verbose 3333@item -v, --verbose, @code{%option verbose} 3334specifies that @code{flex} should write to @file{stderr} a summary of 3335statistics regarding the scanner it generates. Most of the statistics 3336are meaningless to the casual @code{flex} user, but the first line 3337identifies the version of @code{flex} (same as reported by @samp{--version}), 3338and the next line the flags used when generating the scanner, including 3339those that are on by default. 3340 3341 3342 3343@anchor{option-warn} 3344@opindex ---warn 3345@opindex warn 3346@item --warn, @code{%option warn} 3347warn about certain things. In particular, if the default rule can be 3348matched but no default rule has been given, the flex will warn you. 3349We recommend using this option always. 3350 3351@end table 3352 3353@node Miscellaneous Options, , Debugging Options, Scanner Options 3354@section Miscellaneous Options 3355 3356@table @samp 3357@opindex -c 3358@item -c 3359A do-nothing option included for POSIX compliance. 3360 3361@opindex -h 3362@opindex ---help 3363@item -h, -?, --help 3364generates a ``help'' summary of @code{flex}'s options to @file{stdout} 3365and then exits. 3366 3367@opindex -n 3368@item -n 3369Another do-nothing option included for 3370POSIX compliance. 3371 3372@opindex -V 3373@opindex ---version 3374@item -V, --version 3375prints the version number to @file{stdout} and exits. 3376 3377@end table 3378 3379 3380@node Performance, Cxx, Scanner Options, Top 3381@chapter Performance Considerations 3382 3383@cindex performance, considerations 3384The main design goal of @code{flex} is that it generate high-performance 3385scanners. It has been optimized for dealing well with large sets of 3386rules. Aside from the effects on scanner speed of the table compression 3387@samp{-C} options outlined above, there are a number of options/actions 3388which degrade performance. These are, from most expensive to least: 3389 3390@cindex REJECT, performance costs 3391@cindex yylineno, performance costs 3392@cindex trailing context, performance costs 3393@example 3394@verbatim 3395 REJECT 3396 arbitrary trailing context 3397 3398 pattern sets that require backing up 3399 %option yylineno 3400 %array 3401 3402 %option interactive 3403 %option always-interactive 3404 3405 ^ beginning-of-line operator 3406 yymore() 3407@end verbatim 3408@end example 3409 3410with the first two all being quite expensive and the last two being 3411quite cheap. Note also that @code{unput()} is implemented as a routine 3412call that potentially does quite a bit of work, while @code{yyless()} is 3413a quite-cheap macro. So if you are just putting back some excess text 3414you scanned, use @code{yyless()}. 3415 3416@code{REJECT} should be avoided at all costs when performance is 3417important. It is a particularly expensive option. 3418 3419There is one case when @code{%option yylineno} can be expensive. That is when 3420your patterns match long tokens that could @emph{possibly} contain a newline 3421character. There is no performance penalty for rules that can not possibly 3422match newlines, since flex does not need to check them for newlines. In 3423general, you should avoid rules such as @code{[^f]+}, which match very long 3424tokens, including newlines, and may possibly match your entire file! A better 3425approach is to separate @code{[^f]+} into two rules: 3426 3427@example 3428@verbatim 3429%option yylineno 3430%% 3431 [^f\n]+ 3432 \n+ 3433@end verbatim 3434@end example 3435 3436The above scanner does not incur a performance penalty. 3437 3438@cindex patterns, tuning for performance 3439@cindex performance, backing up 3440@cindex backing up, example of eliminating 3441Getting rid of backing up is messy and often may be an enormous amount 3442of work for a complicated scanner. In principal, one begins by using 3443the @samp{-b} flag to generate a @file{lex.backup} file. For example, 3444on the input: 3445 3446@cindex backing up, eliminating 3447@example 3448@verbatim 3449 %% 3450 foo return TOK_KEYWORD; 3451 foobar return TOK_KEYWORD; 3452@end verbatim 3453@end example 3454 3455the file looks like: 3456 3457@example 3458@verbatim 3459 State #6 is non-accepting - 3460 associated rule line numbers: 3461 2 3 3462 out-transitions: [ o ] 3463 jam-transitions: EOF [ \001-n p-\177 ] 3464 3465 State #8 is non-accepting - 3466 associated rule line numbers: 3467 3 3468 out-transitions: [ a ] 3469 jam-transitions: EOF [ \001-` b-\177 ] 3470 3471 State #9 is non-accepting - 3472 associated rule line numbers: 3473 3 3474 out-transitions: [ r ] 3475 jam-transitions: EOF [ \001-q s-\177 ] 3476 3477 Compressed tables always back up. 3478@end verbatim 3479@end example 3480 3481The first few lines tell us that there's a scanner state in which it can 3482make a transition on an 'o' but not on any other character, and that in 3483that state the currently scanned text does not match any rule. The 3484state occurs when trying to match the rules found at lines 2 and 3 in 3485the input file. If the scanner is in that state and then reads 3486something other than an 'o', it will have to back up to find a rule 3487which is matched. With a bit of headscratching one can see that this 3488must be the state it's in when it has seen @samp{fo}. When this has 3489happened, if anything other than another @samp{o} is seen, the scanner 3490will have to back up to simply match the @samp{f} (by the default rule). 3491 3492The comment regarding State #8 indicates there's a problem when 3493@samp{foob} has been scanned. Indeed, on any character other than an 3494@samp{a}, the scanner will have to back up to accept "foo". Similarly, 3495the comment for State #9 concerns when @samp{fooba} has been scanned and 3496an @samp{r} does not follow. 3497 3498The final comment reminds us that there's no point going to all the 3499trouble of removing backing up from the rules unless we're using 3500@samp{-Cf} or @samp{-CF}, since there's no performance gain doing so 3501with compressed scanners. 3502 3503@cindex error rules, to eliminate backing up 3504The way to remove the backing up is to add ``error'' rules: 3505 3506@cindex backing up, eliminating by adding error rules 3507@example 3508@verbatim 3509 %% 3510 foo return TOK_KEYWORD; 3511 foobar return TOK_KEYWORD; 3512 3513 fooba | 3514 foob | 3515 fo { 3516 /* false alarm, not really a keyword */ 3517 return TOK_ID; 3518 } 3519@end verbatim 3520@end example 3521 3522Eliminating backing up among a list of keywords can also be done using a 3523``catch-all'' rule: 3524 3525@cindex backing up, eliminating with catch-all rule 3526@example 3527@verbatim 3528 %% 3529 foo return TOK_KEYWORD; 3530 foobar return TOK_KEYWORD; 3531 3532 [a-z]+ return TOK_ID; 3533@end verbatim 3534@end example 3535 3536This is usually the best solution when appropriate. 3537 3538Backing up messages tend to cascade. With a complicated set of rules 3539it's not uncommon to get hundreds of messages. If one can decipher 3540them, though, it often only takes a dozen or so rules to eliminate the 3541backing up (though it's easy to make a mistake and have an error rule 3542accidentally match a valid token. A possible future @code{flex} feature 3543will be to automatically add rules to eliminate backing up). 3544 3545It's important to keep in mind that you gain the benefits of eliminating 3546backing up only if you eliminate @emph{every} instance of backing up. 3547Leaving just one means you gain nothing. 3548 3549@emph{Variable} trailing context (where both the leading and trailing 3550parts do not have a fixed length) entails almost the same performance 3551loss as @code{REJECT} (i.e., substantial). So when possible a rule 3552like: 3553 3554@cindex trailing context, variable length 3555@example 3556@verbatim 3557 %% 3558 mouse|rat/(cat|dog) run(); 3559@end verbatim 3560@end example 3561 3562is better written: 3563 3564@example 3565@verbatim 3566 %% 3567 mouse/cat|dog run(); 3568 rat/cat|dog run(); 3569@end verbatim 3570@end example 3571 3572or as 3573 3574@example 3575@verbatim 3576 %% 3577 mouse|rat/cat run(); 3578 mouse|rat/dog run(); 3579@end verbatim 3580@end example 3581 3582Note that here the special '|' action does @emph{not} provide any 3583savings, and can even make things worse (@pxref{Limitations}). 3584 3585Another area where the user can increase a scanner's performance (and 3586one that's easier to implement) arises from the fact that the longer the 3587tokens matched, the faster the scanner will run. This is because with 3588long tokens the processing of most input characters takes place in the 3589(short) inner scanning loop, and does not often have to go through the 3590additional work of setting up the scanning environment (e.g., 3591@code{yytext}) for the action. Recall the scanner for C comments: 3592 3593@cindex performance optimization, matching longer tokens 3594@example 3595@verbatim 3596 %x comment 3597 %% 3598 int line_num = 1; 3599 3600 "/*" BEGIN(comment); 3601 3602 <comment>[^*\n]* 3603 <comment>"*"+[^*/\n]* 3604 <comment>\n ++line_num; 3605 <comment>"*"+"/" BEGIN(INITIAL); 3606@end verbatim 3607@end example 3608 3609This could be sped up by writing it as: 3610 3611@example 3612@verbatim 3613 %x comment 3614 %% 3615 int line_num = 1; 3616 3617 "/*" BEGIN(comment); 3618 3619 <comment>[^*\n]* 3620 <comment>[^*\n]*\n ++line_num; 3621 <comment>"*"+[^*/\n]* 3622 <comment>"*"+[^*/\n]*\n ++line_num; 3623 <comment>"*"+"/" BEGIN(INITIAL); 3624@end verbatim 3625@end example 3626 3627Now instead of each newline requiring the processing of another action, 3628recognizing the newlines is distributed over the other rules to keep the 3629matched text as long as possible. Note that @emph{adding} rules does 3630@emph{not} slow down the scanner! The speed of the scanner is 3631independent of the number of rules or (modulo the considerations given 3632at the beginning of this section) how complicated the rules are with 3633regard to operators such as @samp{*} and @samp{|}. 3634 3635@cindex keywords, for performance 3636@cindex performance, using keywords 3637A final example in speeding up a scanner: suppose you want to scan 3638through a file containing identifiers and keywords, one per line 3639and with no other extraneous characters, and recognize all the 3640keywords. A natural first approach is: 3641 3642@cindex performance optimization, recognizing keywords 3643@example 3644@verbatim 3645 %% 3646 asm | 3647 auto | 3648 break | 3649 ... etc ... 3650 volatile | 3651 while /* it's a keyword */ 3652 3653 .|\n /* it's not a keyword */ 3654@end verbatim 3655@end example 3656 3657To eliminate the back-tracking, introduce a catch-all rule: 3658 3659@example 3660@verbatim 3661 %% 3662 asm | 3663 auto | 3664 break | 3665 ... etc ... 3666 volatile | 3667 while /* it's a keyword */ 3668 3669 [a-z]+ | 3670 .|\n /* it's not a keyword */ 3671@end verbatim 3672@end example 3673 3674Now, if it's guaranteed that there's exactly one word per line, then we 3675can reduce the total number of matches by a half by merging in the 3676recognition of newlines with that of the other tokens: 3677 3678@example 3679@verbatim 3680 %% 3681 asm\n | 3682 auto\n | 3683 break\n | 3684 ... etc ... 3685 volatile\n | 3686 while\n /* it's a keyword */ 3687 3688 [a-z]+\n | 3689 .|\n /* it's not a keyword */ 3690@end verbatim 3691@end example 3692 3693One has to be careful here, as we have now reintroduced backing up 3694into the scanner. In particular, while 3695@emph{we} 3696know that there will never be any characters in the input stream 3697other than letters or newlines, 3698@code{flex} 3699can't figure this out, and it will plan for possibly needing to back up 3700when it has scanned a token like @samp{auto} and then the next character 3701is something other than a newline or a letter. Previously it would 3702then just match the @samp{auto} rule and be done, but now it has no @samp{auto} 3703rule, only a @samp{auto\n} rule. To eliminate the possibility of backing up, 3704we could either duplicate all rules but without final newlines, or, 3705since we never expect to encounter such an input and therefore don't 3706how it's classified, we can introduce one more catch-all rule, this 3707one which doesn't include a newline: 3708 3709@example 3710@verbatim 3711 %% 3712 asm\n | 3713 auto\n | 3714 break\n | 3715 ... etc ... 3716 volatile\n | 3717 while\n /* it's a keyword */ 3718 3719 [a-z]+\n | 3720 [a-z]+ | 3721 .|\n /* it's not a keyword */ 3722@end verbatim 3723@end example 3724 3725Compiled with @samp{-Cf}, this is about as fast as one can get a 3726@code{flex} scanner to go for this particular problem. 3727 3728A final note: @code{flex} is slow when matching @code{NUL}s, 3729particularly when a token contains multiple @code{NUL}s. It's best to 3730write rules which match @emph{short} amounts of text if it's anticipated 3731that the text will often include @code{NUL}s. 3732 3733Another final note regarding performance: as mentioned in 3734@ref{Matching}, dynamically resizing @code{yytext} to accommodate huge 3735tokens is a slow process because it presently requires that the (huge) 3736token be rescanned from the beginning. Thus if performance is vital, 3737you should attempt to match ``large'' quantities of text but not 3738``huge'' quantities, where the cutoff between the two is at about 8K 3739characters per token. 3740 3741@node Cxx, Reentrant, Performance, Top 3742@chapter Generating C++ Scanners 3743 3744@cindex c++, experimental form of scanner class 3745@cindex experimental form of c++ scanner class 3746@strong{IMPORTANT}: the present form of the scanning class is @emph{experimental} 3747and may change considerably between major releases. 3748 3749@cindex C++ 3750@cindex member functions, C++ 3751@cindex methods, c++ 3752@code{flex} provides two different ways to generate scanners for use 3753with C++. The first way is to simply compile a scanner generated by 3754@code{flex} using a C++ compiler instead of a C compiler. You should 3755not encounter any compilation errors (@pxref{Reporting Bugs}). You can 3756then use C++ code in your rule actions instead of C code. Note that the 3757default input source for your scanner remains @file{yyin}, and default 3758echoing is still done to @file{yyout}. Both of these remain @code{FILE 3759*} variables and not C++ @emph{streams}. 3760 3761You can also use @code{flex} to generate a C++ scanner class, using the 3762@samp{-+} option (or, equivalently, @code{%option c++)}, which is 3763automatically specified if the name of the @code{flex} executable ends 3764in a '+', such as @code{flex++}. When using this option, @code{flex} 3765defaults to generating the scanner to the file @file{lex.yy.cc} instead 3766of @file{lex.yy.c}. The generated scanner includes the header file 3767@file{FlexLexer.h}, which defines the interface to two C++ classes. 3768 3769The first class, 3770@code{FlexLexer}, 3771provides an abstract base class defining the general scanner class 3772interface. It provides the following member functions: 3773 3774@table @code 3775@findex YYText (C++ only) 3776@item const char* YYText() 3777returns the text of the most recently matched token, the equivalent of 3778@code{yytext}. 3779 3780@findex YYLeng (C++ only) 3781@item int YYLeng() 3782returns the length of the most recently matched token, the equivalent of 3783@code{yyleng}. 3784 3785@findex lineno (C++ only) 3786@item int lineno() const 3787returns the current input line number (see @code{%option yylineno)}, or 3788@code{1} if @code{%option yylineno} was not used. 3789 3790@findex set_debug (C++ only) 3791@item void set_debug( int flag ) 3792sets the debugging flag for the scanner, equivalent to assigning to 3793@code{yy_flex_debug} (@pxref{Scanner Options}). Note that you must build 3794the scanner using @code{%option debug} to include debugging information 3795in it. 3796 3797@findex debug (C++ only) 3798@item int debug() const 3799returns the current setting of the debugging flag. 3800@end table 3801 3802Also provided are member functions equivalent to 3803@code{yy_switch_to_buffer()}, @code{yy_create_buffer()} (though the 3804first argument is an @code{istream*} object pointer and not a 3805@code{FILE*)}, @code{yy_flush_buffer()}, @code{yy_delete_buffer()}, and 3806@code{yyrestart()} (again, the first argument is a @code{istream*} 3807object pointer). 3808 3809@tindex yyFlexLexer (C++ only) 3810@tindex FlexLexer (C++ only) 3811The second class defined in @file{FlexLexer.h} is @code{yyFlexLexer}, 3812which is derived from @code{FlexLexer}. It defines the following 3813additional member functions: 3814 3815@table @code 3816@findex yyFlexLexer constructor (C++ only) 3817@item yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 ) 3818constructs a @code{yyFlexLexer} object using the given streams for input 3819and output. If not specified, the streams default to @code{cin} and 3820@code{cout}, respectively. 3821 3822@findex yylex (C++ version) 3823@item virtual int yylex() 3824performs the same role is @code{yylex()} does for ordinary @code{flex} 3825scanners: it scans the input stream, consuming tokens, until a rule's 3826action returns a value. If you derive a subclass @code{S} from 3827@code{yyFlexLexer} and want to access the member functions and variables 3828of @code{S} inside @code{yylex()}, then you need to use @code{%option 3829yyclass="S"} to inform @code{flex} that you will be using that subclass 3830instead of @code{yyFlexLexer}. In this case, rather than generating 3831@code{yyFlexLexer::yylex()}, @code{flex} generates @code{S::yylex()} 3832(and also generates a dummy @code{yyFlexLexer::yylex()} that calls 3833@code{yyFlexLexer::LexerError()} if called). 3834 3835@findex switch_streams (C++ only) 3836@item virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0) 3837reassigns @code{yyin} to @code{new_in} (if non-null) and @code{yyout} to 3838@code{new_out} (if non-null), deleting the previous input buffer if 3839@code{yyin} is reassigned. 3840 3841@item int yylex( istream* new_in, ostream* new_out = 0 ) 3842first switches the input streams via @code{switch_streams( new_in, 3843new_out )} and then returns the value of @code{yylex()}. 3844@end table 3845 3846In addition, @code{yyFlexLexer} defines the following protected virtual 3847functions which you can redefine in derived classes to tailor the 3848scanner: 3849 3850@table @code 3851@findex LexerInput (C++ only) 3852@item virtual int LexerInput( char* buf, int max_size ) 3853reads up to @code{max_size} characters into @code{buf} and returns the 3854number of characters read. To indicate end-of-input, return 0 3855characters. Note that @code{interactive} scanners (see the @samp{-B} 3856and @samp{-I} flags in @ref{Scanner Options}) define the macro 3857@code{YY_INTERACTIVE}. If you redefine @code{LexerInput()} and need to 3858take different actions depending on whether or not the scanner might be 3859scanning an interactive input source, you can test for the presence of 3860this name via @code{#ifdef} statements. 3861 3862@findex LexerOutput (C++ only) 3863@item virtual void LexerOutput( const char* buf, int size ) 3864writes out @code{size} characters from the buffer @code{buf}, which, while 3865@code{NUL}-terminated, may also contain internal @code{NUL}s if the 3866scanner's rules can match text with @code{NUL}s in them. 3867 3868@cindex error reporting, in C++ 3869@findex LexerError (C++ only) 3870@item virtual void LexerError( const char* msg ) 3871reports a fatal error message. The default version of this function 3872writes the message to the stream @code{cerr} and exits. 3873@end table 3874 3875Note that a @code{yyFlexLexer} object contains its @emph{entire} 3876scanning state. Thus you can use such objects to create reentrant 3877scanners, but see also @ref{Reentrant}. You can instantiate multiple 3878instances of the same @code{yyFlexLexer} class, and you can also combine 3879multiple C++ scanner classes together in the same program using the 3880@samp{-P} option discussed above. 3881 3882Finally, note that the @code{%array} feature is not available to C++ 3883scanner classes; you must use @code{%pointer} (the default). 3884 3885Here is an example of a simple C++ scanner: 3886 3887@cindex C++ scanners, use of 3888@example 3889@verbatim 3890 // An example of using the flex C++ scanner class. 3891 3892 %{ 3893 #include <iostream> 3894 using namespace std; 3895 int mylineno = 0; 3896 %} 3897 3898 %option noyywrap 3899 3900 string \"[^\n"]+\" 3901 3902 ws [ \t]+ 3903 3904 alpha [A-Za-z] 3905 dig [0-9] 3906 name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])* 3907 num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)? 3908 num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)? 3909 number {num1}|{num2} 3910 3911 %% 3912 3913 {ws} /* skip blanks and tabs */ 3914 3915 "/*" { 3916 int c; 3917 3918 while((c = yyinput()) != 0) 3919 { 3920 if(c == '\n') 3921 ++mylineno; 3922 3923 else if(c == '*') 3924 { 3925 if((c = yyinput()) == '/') 3926 break; 3927 else 3928 unput(c); 3929 } 3930 } 3931 } 3932 3933 {number} cout << "number " << YYText() << '\n'; 3934 3935 \n mylineno++; 3936 3937 {name} cout << "name " << YYText() << '\n'; 3938 3939 {string} cout << "string " << YYText() << '\n'; 3940 3941 %% 3942 3943 int main( int /* argc */, char** /* argv */ ) 3944 { 3945 FlexLexer* lexer = new yyFlexLexer; 3946 while(lexer->yylex() != 0) 3947 ; 3948 return 0; 3949 } 3950@end verbatim 3951@end example 3952 3953@cindex C++, multiple different scanners 3954If you want to create multiple (different) lexer classes, you use the 3955@samp{-P} flag (or the @code{prefix=} option) to rename each 3956@code{yyFlexLexer} to some other @samp{xxFlexLexer}. You then can 3957include @file{<FlexLexer.h>} in your other sources once per lexer class, 3958first renaming @code{yyFlexLexer} as follows: 3959 3960@cindex include files, with C++ 3961@cindex header files, with C++ 3962@cindex C++ scanners, including multiple scanners 3963@example 3964@verbatim 3965 #undef yyFlexLexer 3966 #define yyFlexLexer xxFlexLexer 3967 #include <FlexLexer.h> 3968 3969 #undef yyFlexLexer 3970 #define yyFlexLexer zzFlexLexer 3971 #include <FlexLexer.h> 3972@end verbatim 3973@end example 3974 3975if, for example, you used @code{%option prefix="xx"} for one of your 3976scanners and @code{%option prefix="zz"} for the other. 3977 3978@node Reentrant, Lex and Posix, Cxx, Top 3979@chapter Reentrant C Scanners 3980 3981@cindex reentrant, explanation 3982@code{flex} has the ability to generate a reentrant C scanner. This is 3983accomplished by specifying @code{%option reentrant} (@samp{-R}) The generated 3984scanner is both portable, and safe to use in one or more separate threads of 3985control. The most common use for reentrant scanners is from within 3986multi-threaded applications. Any thread may create and execute a reentrant 3987@code{flex} scanner without the need for synchronization with other threads. 3988 3989@menu 3990* Reentrant Uses:: 3991* Reentrant Overview:: 3992* Reentrant Example:: 3993* Reentrant Detail:: 3994* Reentrant Functions:: 3995@end menu 3996 3997@node Reentrant Uses, Reentrant Overview, Reentrant, Reentrant 3998@section Uses for Reentrant Scanners 3999 4000However, there are other uses for a reentrant scanner. For example, you 4001could scan two or more files simultaneously to implement a @code{diff} at 4002the token level (i.e., instead of at the character level): 4003 4004@cindex reentrant scanners, multiple interleaved scanners 4005@example 4006@verbatim 4007 /* Example of maintaining more than one active scanner. */ 4008 4009 do { 4010 int tok1, tok2; 4011 4012 tok1 = yylex( scanner_1 ); 4013 tok2 = yylex( scanner_2 ); 4014 4015 if( tok1 != tok2 ) 4016 printf("Files are different."); 4017 4018 } while ( tok1 && tok2 ); 4019@end verbatim 4020@end example 4021 4022Another use for a reentrant scanner is recursion. 4023(Note that a recursive scanner can also be created using a non-reentrant scanner and 4024buffer states. @xref{Multiple Input Buffers}.) 4025 4026The following crude scanner supports the @samp{eval} command by invoking 4027another instance of itself. 4028 4029@cindex reentrant scanners, recursive invocation 4030@example 4031@verbatim 4032 /* Example of recursive invocation. */ 4033 4034 %option reentrant 4035 4036 %% 4037 "eval(".+")" { 4038 yyscan_t scanner; 4039 YY_BUFFER_STATE buf; 4040 4041 yylex_init( &scanner ); 4042 yytext[yyleng-1] = ' '; 4043 4044 buf = yy_scan_string( yytext + 5, scanner ); 4045 yylex( scanner ); 4046 4047 yy_delete_buffer(buf,scanner); 4048 yylex_destroy( scanner ); 4049 } 4050 ... 4051 %% 4052@end verbatim 4053@end example 4054 4055@node Reentrant Overview, Reentrant Example, Reentrant Uses, Reentrant 4056@section An Overview of the Reentrant API 4057 4058@cindex reentrant, API explanation 4059The API for reentrant scanners is different than for non-reentrant 4060scanners. Here is a quick overview of the API: 4061 4062@itemize 4063@code{%option reentrant} must be specified. 4064 4065@item 4066All functions take one additional argument: @code{yyscanner} 4067 4068@item 4069All global variables are replaced by their macro equivalents. 4070(We tell you this because it may be important to you during debugging.) 4071 4072@item 4073@code{yylex_init} and @code{yylex_destroy} must be called before and 4074after @code{yylex}, respectively. 4075 4076@item 4077Accessor methods (get/set functions) provide access to common 4078@code{flex} variables. 4079 4080@item 4081User-specific data can be stored in @code{yyextra}. 4082@end itemize 4083 4084@node Reentrant Example, Reentrant Detail, Reentrant Overview, Reentrant 4085@section Reentrant Example 4086 4087First, an example of a reentrant scanner: 4088@cindex reentrant, example of 4089@example 4090@verbatim 4091 /* This scanner prints "//" comments. */ 4092 4093 %option reentrant stack noyywrap 4094 %x COMMENT 4095 4096 %% 4097 4098 "//" yy_push_state( COMMENT, yyscanner); 4099 .|\n 4100 4101 <COMMENT>\n yy_pop_state( yyscanner ); 4102 <COMMENT>[^\n]+ fprintf( yyout, "%s\n", yytext); 4103 4104 %% 4105 4106 int main ( int argc, char * argv[] ) 4107 { 4108 yyscan_t scanner; 4109 4110 yylex_init ( &scanner ); 4111 yylex ( scanner ); 4112 yylex_destroy ( scanner ); 4113 return 0; 4114 } 4115@end verbatim 4116@end example 4117 4118@node Reentrant Detail, Reentrant Functions, Reentrant Example, Reentrant 4119@section The Reentrant API in Detail 4120 4121Here are the things you need to do or know to use the reentrant C API of 4122@code{flex}. 4123 4124@menu 4125* Specify Reentrant:: 4126* Extra Reentrant Argument:: 4127* Global Replacement:: 4128* Init and Destroy Functions:: 4129* Accessor Methods:: 4130* Extra Data:: 4131* About yyscan_t:: 4132@end menu 4133 4134@node Specify Reentrant, Extra Reentrant Argument, Reentrant Detail, Reentrant Detail 4135@subsection Declaring a Scanner As Reentrant 4136 4137 %option reentrant (--reentrant) must be specified. 4138 4139Notice that @code{%option reentrant} is specified in the above example 4140(@pxref{Reentrant Example}. Had this option not been specified, 4141@code{flex} would have happily generated a non-reentrant scanner without 4142complaining. You may explicitly specify @code{%option noreentrant}, if 4143you do @emph{not} want a reentrant scanner, although it is not 4144necessary. The default is to generate a non-reentrant scanner. 4145 4146@node Extra Reentrant Argument, Global Replacement, Specify Reentrant, Reentrant Detail 4147@subsection The Extra Argument 4148 4149@cindex reentrant, calling functions 4150@vindex yyscanner (reentrant only) 4151All functions take one additional argument: @code{yyscanner}. 4152 4153Notice that the calls to @code{yy_push_state} and @code{yy_pop_state} 4154both have an argument, @code{yyscanner} , that is not present in a 4155non-reentrant scanner. Here are the declarations of 4156@code{yy_push_state} and @code{yy_pop_state} in the reentrant scanner: 4157 4158@example 4159@verbatim 4160 static void yy_push_state ( int new_state , yyscan_t yyscanner ) ; 4161 static void yy_pop_state ( yyscan_t yyscanner ) ; 4162@end verbatim 4163@end example 4164 4165Notice that the argument @code{yyscanner} appears in the declaration of 4166both functions. In fact, all @code{flex} functions in a reentrant 4167scanner have this additional argument. It is always the last argument 4168in the argument list, it is always of type @code{yyscan_t} (which is 4169typedef'd to @code{void *}) and it is 4170always named @code{yyscanner}. As you may have guessed, 4171@code{yyscanner} is a pointer to an opaque data structure encapsulating 4172the current state of the scanner. For a list of function declarations, 4173see @ref{Reentrant Functions}. Note that preprocessor macros, such as 4174@code{BEGIN}, @code{ECHO}, and @code{REJECT}, do not take this 4175additional argument. 4176 4177@node Global Replacement, Init and Destroy Functions, Extra Reentrant Argument, Reentrant Detail 4178@subsection Global Variables Replaced By Macros 4179 4180@cindex reentrant, accessing flex variables 4181All global variables in traditional flex have been replaced by macro equivalents. 4182 4183Note that in the above example, @code{yyout} and @code{yytext} are 4184not plain variables. These are macros that will expand to their equivalent lvalue. 4185All of the familiar @code{flex} globals have been replaced by their macro 4186equivalents. In particular, @code{yytext}, @code{yyleng}, @code{yylineno}, 4187@code{yyin}, @code{yyout}, @code{yyextra}, @code{yylval}, and @code{yylloc} 4188are macros. You may safely use these macros in actions as if they were plain 4189variables. We only tell you this so you don't expect to link to these variables 4190externally. Currently, each macro expands to a member of an internal struct, e.g., 4191 4192@example 4193@verbatim 4194#define yytext (((struct yyguts_t*)yyscanner)->yytext_r) 4195@end verbatim 4196@end example 4197 4198One important thing to remember about 4199@code{yytext} 4200and friends is that 4201@code{yytext} 4202is not a global variable in a reentrant 4203scanner, you can not access it directly from outside an action or from 4204other functions. You must use an accessor method, e.g., 4205@code{yyget_text}, 4206to accomplish this. (See below). 4207 4208@node Init and Destroy Functions, Accessor Methods, Global Replacement, Reentrant Detail 4209@subsection Init and Destroy Functions 4210 4211@cindex memory, considerations for reentrant scanners 4212@cindex reentrant, initialization 4213@findex yylex_init 4214@findex yylex_destroy 4215 4216@code{yylex_init} and @code{yylex_destroy} must be called before and 4217after @code{yylex}, respectively. 4218 4219@example 4220@verbatim 4221 int yylex_init ( yyscan_t * ptr_yy_globals ) ; 4222 int yylex_init_extra ( YY_EXTRA_TYPE user_defined, yyscan_t * ptr_yy_globals ) ; 4223 int yylex ( yyscan_t yyscanner ) ; 4224 int yylex_destroy ( yyscan_t yyscanner ) ; 4225@end verbatim 4226@end example 4227 4228The function @code{yylex_init} must be called before calling any other 4229function. The argument to @code{yylex_init} is the address of an 4230uninitialized pointer to be filled in by @code{yylex_init}, overwriting 4231any previous contents. The function @code{yylex_init_extra} may be used 4232instead, taking as its first argument a variable of type @code{YY_EXTRA_TYPE}. 4233See the section on yyextra, below, for more details. 4234 4235The value stored in @code{ptr_yy_globals} should 4236thereafter be passed to @code{yylex} and @code{yylex_destroy}. Flex 4237does not save the argument passed to @code{yylex_init}, so it is safe to 4238pass the address of a local pointer to @code{yylex_init} so long as it remains 4239in scope for the duration of all calls to the scanner, up to and including 4240the call to @code{yylex_destroy}. 4241 4242The function 4243@code{yylex} should be familiar to you by now. The reentrant version 4244takes one argument, which is the value returned (via an argument) by 4245@code{yylex_init}. Otherwise, it behaves the same as the non-reentrant 4246version of @code{yylex}. 4247 4248Both @code{yylex_init} and @code{yylex_init_extra} returns 0 (zero) on success, 4249or non-zero on failure, in which case errno is set to one of the following values: 4250 4251@itemize 4252@item ENOMEM 4253Memory allocation error. @xref{memory-management}. 4254@item EINVAL 4255Invalid argument. 4256@end itemize 4257 4258 4259The function @code{yylex_destroy} should be 4260called to free resources used by the scanner. After @code{yylex_destroy} 4261is called, the contents of @code{yyscanner} should not be used. Of 4262course, there is no need to destroy a scanner if you plan to reuse it. 4263A @code{flex} scanner (both reentrant and non-reentrant) may be 4264restarted by calling @code{yyrestart}. 4265 4266Below is an example of a program that creates a scanner, uses it, then destroys 4267it when done: 4268 4269@example 4270@verbatim 4271 int main () 4272 { 4273 yyscan_t scanner; 4274 int tok; 4275 4276 yylex_init(&scanner); 4277 4278 while ((tok=yylex(scanner)) > 0) 4279 printf("tok=%d yytext=%s\n", tok, yyget_text(scanner)); 4280 4281 yylex_destroy(scanner); 4282 return 0; 4283 } 4284@end verbatim 4285@end example 4286 4287@node Accessor Methods, Extra Data, Init and Destroy Functions, Reentrant Detail 4288@subsection Accessing Variables with Reentrant Scanners 4289 4290@cindex reentrant, accessor functions 4291Accessor methods (get/set functions) provide access to common 4292@code{flex} variables. 4293 4294Many scanners that you build will be part of a larger project. Portions 4295of your project will need access to @code{flex} values, such as 4296@code{yytext}. In a non-reentrant scanner, these values are global, so 4297there is no problem accessing them. However, in a reentrant scanner, there are no 4298global @code{flex} values. You can not access them directly. Instead, 4299you must access @code{flex} values using accessor methods (get/set 4300functions). Each accessor method is named @code{yyget_NAME} or 4301@code{yyset_NAME}, where @code{NAME} is the name of the @code{flex} 4302variable you want. For example: 4303 4304@cindex accessor functions, use of 4305@example 4306@verbatim 4307 /* Set the last character of yytext to NULL. */ 4308 void chop ( yyscan_t scanner ) 4309 { 4310 int len = yyget_leng( scanner ); 4311 yyget_text( scanner )[len - 1] = '\0'; 4312 } 4313@end verbatim 4314@end example 4315 4316The above code may be called from within an action like this: 4317 4318@example 4319@verbatim 4320 %% 4321 .+\n { chop( yyscanner );} 4322@end verbatim 4323@end example 4324 4325You may find that @code{%option header-file} is particularly useful for generating 4326prototypes of all the accessor functions. @xref{option-header}. 4327 4328@node Extra Data, About yyscan_t, Accessor Methods, Reentrant Detail 4329@subsection Extra Data 4330 4331@cindex reentrant, extra data 4332@vindex yyextra 4333User-specific data can be stored in @code{yyextra}. 4334 4335In a reentrant scanner, it is unwise to use global variables to 4336communicate with or maintain state between different pieces of your program. 4337However, you may need access to external data or invoke external functions 4338from within the scanner actions. 4339Likewise, you may need to pass information to your scanner 4340(e.g., open file descriptors, or database connections). 4341In a non-reentrant scanner, the only way to do this would be through the 4342use of global variables. 4343@code{Flex} allows you to store arbitrary, ``extra'' data in a scanner. 4344This data is accessible through the accessor methods 4345@code{yyget_extra} and @code{yyset_extra} 4346from outside the scanner, and through the shortcut macro 4347@code{yyextra} 4348from within the scanner itself. They are defined as follows: 4349 4350@tindex YY_EXTRA_TYPE (reentrant only) 4351@findex yyget_extra 4352@findex yyset_extra 4353@example 4354@verbatim 4355 #define YY_EXTRA_TYPE void* 4356 YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); 4357 void yyset_extra ( YY_EXTRA_TYPE arbitrary_data , yyscan_t scanner); 4358@end verbatim 4359@end example 4360 4361In addition, an extra form of @code{yylex_init} is provided, 4362@code{yylex_init_extra}. This function is provided so that the yyextra value can 4363be accessed from within the very first yyalloc, used to allocate 4364the scanner itself. 4365 4366By default, @code{YY_EXTRA_TYPE} is defined as type @code{void *}. You 4367may redefine this type using @code{%option extra-type="your_type"} in 4368the scanner: 4369 4370@cindex YY_EXTRA_TYPE, defining your own type 4371@example 4372@verbatim 4373 /* An example of overriding YY_EXTRA_TYPE. */ 4374 %{ 4375 #include <sys/stat.h> 4376 #include <unistd.h> 4377 %} 4378 %option reentrant 4379 %option extra-type="struct stat *" 4380 %% 4381 4382 __filesize__ printf( "%ld", yyextra->st_size ); 4383 __lastmod__ printf( "%ld", yyextra->st_mtime ); 4384 %% 4385 void scan_file( char* filename ) 4386 { 4387 yyscan_t scanner; 4388 struct stat buf; 4389 FILE *in; 4390 4391 in = fopen( filename, "r" ); 4392 stat( filename, &buf ); 4393 4394 yylex_init_extra( buf, &scanner ); 4395 yyset_in( in, scanner ); 4396 yylex( scanner ); 4397 yylex_destroy( scanner ); 4398 4399 fclose( in ); 4400 } 4401@end verbatim 4402@end example 4403 4404 4405@node About yyscan_t, , Extra Data, Reentrant Detail 4406@subsection About yyscan_t 4407 4408@tindex yyscan_t (reentrant only) 4409@code{yyscan_t} is defined as: 4410 4411@example 4412@verbatim 4413 typedef void* yyscan_t; 4414@end verbatim 4415@end example 4416 4417It is initialized by @code{yylex_init()} to point to 4418an internal structure. You should never access this value 4419directly. In particular, you should never attempt to free it 4420(use @code{yylex_destroy()} instead.) 4421 4422@node Reentrant Functions, , Reentrant Detail, Reentrant 4423@section Functions and Macros Available in Reentrant C Scanners 4424 4425The following Functions are available in a reentrant scanner: 4426 4427@findex yyget_text 4428@findex yyget_leng 4429@findex yyget_in 4430@findex yyget_out 4431@findex yyget_lineno 4432@findex yyset_in 4433@findex yyset_out 4434@findex yyset_lineno 4435@findex yyget_debug 4436@findex yyset_debug 4437@findex yyget_extra 4438@findex yyset_extra 4439 4440@example 4441@verbatim 4442 char *yyget_text ( yyscan_t scanner ); 4443 int yyget_leng ( yyscan_t scanner ); 4444 FILE *yyget_in ( yyscan_t scanner ); 4445 FILE *yyget_out ( yyscan_t scanner ); 4446 int yyget_lineno ( yyscan_t scanner ); 4447 YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); 4448 int yyget_debug ( yyscan_t scanner ); 4449 4450 void yyset_debug ( int flag, yyscan_t scanner ); 4451 void yyset_in ( FILE * in_str , yyscan_t scanner ); 4452 void yyset_out ( FILE * out_str , yyscan_t scanner ); 4453 void yyset_lineno ( int line_number , yyscan_t scanner ); 4454 void yyset_extra ( YY_EXTRA_TYPE user_defined , yyscan_t scanner ); 4455@end verbatim 4456@end example 4457 4458There are no ``set'' functions for yytext and yyleng. This is intentional. 4459 4460The following Macro shortcuts are available in actions in a reentrant 4461scanner: 4462 4463@example 4464@verbatim 4465 yytext 4466 yyleng 4467 yyin 4468 yyout 4469 yylineno 4470 yyextra 4471 yy_flex_debug 4472@end verbatim 4473@end example 4474 4475@cindex yylineno, in a reentrant scanner 4476In a reentrant C scanner, support for yylineno is always present 4477(i.e., you may access yylineno), but the value is never modified by 4478@code{flex} unless @code{%option yylineno} is enabled. This is to allow 4479the user to maintain the line count independently of @code{flex}. 4480 4481@anchor{bison-functions} 4482The following functions and macros are made available when @code{%option 4483bison-bridge} (@samp{--bison-bridge}) is specified: 4484 4485@example 4486@verbatim 4487 YYSTYPE * yyget_lval ( yyscan_t scanner ); 4488 void yyset_lval ( YYSTYPE * yylvalp , yyscan_t scanner ); 4489 yylval 4490@end verbatim 4491@end example 4492 4493The following functions and macros are made available 4494when @code{%option bison-locations} (@samp{--bison-locations}) is specified: 4495 4496@example 4497@verbatim 4498 YYLTYPE *yyget_lloc ( yyscan_t scanner ); 4499 void yyset_lloc ( YYLTYPE * yyllocp , yyscan_t scanner ); 4500 yylloc 4501@end verbatim 4502@end example 4503 4504Support for yylval assumes that @code{YYSTYPE} is a valid type. Support for 4505yylloc assumes that @code{YYSLYPE} is a valid type. Typically, these types are 4506generated by @code{bison}, and are included in section 1 of the @code{flex} 4507input. 4508 4509@node Lex and Posix, Memory Management, Reentrant, Top 4510@chapter Incompatibilities with Lex and Posix 4511 4512@cindex POSIX and lex 4513@cindex lex (traditional) and POSIX 4514 4515@code{flex} is a rewrite of the AT&T Unix @emph{lex} tool (the two 4516implementations do not share any code, though), with some extensions and 4517incompatibilities, both of which are of concern to those who wish to 4518write scanners acceptable to both implementations. @code{flex} is fully 4519compliant with the POSIX @code{lex} specification, except that when 4520using @code{%pointer} (the default), a call to @code{unput()} destroys 4521the contents of @code{yytext}, which is counter to the POSIX 4522specification. In this section we discuss all of the known areas of 4523incompatibility between @code{flex}, AT&T @code{lex}, and the POSIX 4524specification. @code{flex}'s @samp{-l} option turns on maximum 4525compatibility with the original AT&T @code{lex} implementation, at the 4526cost of a major loss in the generated scanner's performance. We note 4527below which incompatibilities can be overcome using the @samp{-l} 4528option. @code{flex} is fully compatible with @code{lex} with the 4529following exceptions: 4530 4531@itemize 4532@item 4533The undocumented @code{lex} scanner internal variable @code{yylineno} is 4534not supported unless @samp{-l} or @code{%option yylineno} is used. 4535 4536@item 4537@code{yylineno} should be maintained on a per-buffer basis, rather than 4538a per-scanner (single global variable) basis. 4539 4540@item 4541@code{yylineno} is not part of the POSIX specification. 4542 4543@item 4544The @code{input()} routine is not redefinable, though it may be called 4545to read characters following whatever has been matched by a rule. If 4546@code{input()} encounters an end-of-file the normal @code{yywrap()} 4547processing is done. A ``real'' end-of-file is returned by 4548@code{input()} as @code{EOF}. 4549 4550@item 4551Input is instead controlled by defining the @code{YY_INPUT()} macro. 4552 4553@item 4554The @code{flex} restriction that @code{input()} cannot be redefined is 4555in accordance with the POSIX specification, which simply does not 4556specify any way of controlling the scanner's input other than by making 4557an initial assignment to @file{yyin}. 4558 4559@item 4560The @code{unput()} routine is not redefinable. This restriction is in 4561accordance with POSIX. 4562 4563@item 4564@code{flex} scanners are not as reentrant as @code{lex} scanners. In 4565particular, if you have an interactive scanner and an interrupt handler 4566which long-jumps out of the scanner, and the scanner is subsequently 4567called again, you may get the following message: 4568 4569@cindex error messages, end of buffer missed 4570@example 4571@verbatim 4572 fatal flex scanner internal error--end of buffer missed 4573@end verbatim 4574@end example 4575 4576To reenter the scanner, first use: 4577 4578@cindex restarting the scanner 4579@example 4580@verbatim 4581 yyrestart( yyin ); 4582@end verbatim 4583@end example 4584 4585Note that this call will throw away any buffered input; usually this 4586isn't a problem with an interactive scanner. @xref{Reentrant}, for 4587@code{flex}'s reentrant API. 4588 4589@item 4590Also note that @code{flex} C++ scanner classes 4591@emph{are} 4592reentrant, so if using C++ is an option for you, you should use 4593them instead. @xref{Cxx}, and @ref{Reentrant} for details. 4594 4595@item 4596@code{output()} is not supported. Output from the @b{ECHO} macro is 4597done to the file-pointer @code{yyout} (default @file{stdout)}. 4598 4599@item 4600@code{output()} is not part of the POSIX specification. 4601 4602@item 4603@code{lex} does not support exclusive start conditions (%x), though they 4604are in the POSIX specification. 4605 4606@item 4607When definitions are expanded, @code{flex} encloses them in parentheses. 4608With @code{lex}, the following: 4609 4610@cindex name definitions, not POSIX 4611@example 4612@verbatim 4613 NAME [A-Z][A-Z0-9]* 4614 %% 4615 foo{NAME}? printf( "Found it\n" ); 4616 %% 4617@end verbatim 4618@end example 4619 4620will not match the string @samp{foo} because when the macro is expanded 4621the rule is equivalent to @samp{foo[A-Z][A-Z0-9]*?} and the precedence 4622is such that the @samp{?} is associated with @samp{[A-Z0-9]*}. With 4623@code{flex}, the rule will be expanded to @samp{foo([A-Z][A-Z0-9]*)?} 4624and so the string @samp{foo} will match. 4625 4626@item 4627Note that if the definition begins with @samp{^} or ends with @samp{$} 4628then it is @emph{not} expanded with parentheses, to allow these 4629operators to appear in definitions without losing their special 4630meanings. But the @samp{<s>}, @samp{/}, and @code{<<EOF>>} operators 4631cannot be used in a @code{flex} definition. 4632 4633@item 4634Using @samp{-l} results in the @code{lex} behavior of no parentheses 4635around the definition. 4636 4637@item 4638The POSIX specification is that the definition be enclosed in parentheses. 4639 4640@item 4641Some implementations of @code{lex} allow a rule's action to begin on a 4642separate line, if the rule's pattern has trailing whitespace: 4643 4644@cindex patterns and actions on different lines 4645@example 4646@verbatim 4647 %% 4648 foo|bar<space here> 4649 { foobar_action();} 4650@end verbatim 4651@end example 4652 4653@code{flex} does not support this feature. 4654 4655@item 4656The @code{lex} @code{%r} (generate a Ratfor scanner) option is not 4657supported. It is not part of the POSIX specification. 4658 4659@item 4660After a call to @code{unput()}, @emph{yytext} is undefined until the 4661next token is matched, unless the scanner was built using @code{%array}. 4662This is not the case with @code{lex} or the POSIX specification. The 4663@samp{-l} option does away with this incompatibility. 4664 4665@item 4666The precedence of the @samp{@{,@}} (numeric range) operator is 4667different. The AT&T and POSIX specifications of @code{lex} 4668interpret @samp{abc@{1,3@}} as match one, two, 4669or three occurrences of @samp{abc}'', whereas @code{flex} interprets it 4670as ``match @samp{ab} followed by one, two, or three occurrences of 4671@samp{c}''. The @samp{-l} and @samp{--posix} options do away with this 4672incompatibility. 4673 4674@item 4675The precedence of the @samp{^} operator is different. @code{lex} 4676interprets @samp{^foo|bar} as ``match either 'foo' at the beginning of a 4677line, or 'bar' anywhere'', whereas @code{flex} interprets it as ``match 4678either @samp{foo} or @samp{bar} if they come at the beginning of a 4679line''. The latter is in agreement with the POSIX specification. 4680 4681@item 4682The special table-size declarations such as @code{%a} supported by 4683@code{lex} are not required by @code{flex} scanners.. @code{flex} 4684ignores them. 4685@item 4686The name @code{FLEX_SCANNER} is @code{#define}'d so scanners may be 4687written for use with either @code{flex} or @code{lex}. Scanners also 4688include @code{YY_FLEX_MAJOR_VERSION}, @code{YY_FLEX_MINOR_VERSION} 4689and @code{YY_FLEX_SUBMINOR_VERSION} 4690indicating which version of @code{flex} generated the scanner. For 4691example, for the 2.5.22 release, these defines would be 2, 5 and 22 4692respectively. If the version of @code{flex} being used is a beta 4693version, then the symbol @code{FLEX_BETA} is defined. 4694 4695@item 4696The symbols @samp{[[} and @samp{]]} in the code sections of the input 4697may conflict with the m4 delimiters. @xref{M4 Dependency}. 4698 4699 4700@end itemize 4701 4702@cindex POSIX comp;compliance 4703@cindex non-POSIX features of flex 4704The following @code{flex} features are not included in @code{lex} or the 4705POSIX specification: 4706 4707@itemize 4708@item 4709C++ scanners 4710@item 4711%option 4712@item 4713start condition scopes 4714@item 4715start condition stacks 4716@item 4717interactive/non-interactive scanners 4718@item 4719yy_scan_string() and friends 4720@item 4721yyterminate() 4722@item 4723yy_set_interactive() 4724@item 4725yy_set_bol() 4726@item 4727YY_AT_BOL() 4728 <<EOF>> 4729@item 4730<*> 4731@item 4732YY_DECL 4733@item 4734YY_START 4735@item 4736YY_USER_ACTION 4737@item 4738YY_USER_INIT 4739@item 4740#line directives 4741@item 4742%@{@}'s around actions 4743@item 4744reentrant C API 4745@item 4746multiple actions on a line 4747@item 4748almost all of the @code{flex} command-line options 4749@end itemize 4750 4751The feature ``multiple actions on a line'' 4752refers to the fact that with @code{flex} you can put multiple actions on 4753the same line, separated with semi-colons, while with @code{lex}, the 4754following: 4755 4756@example 4757@verbatim 4758 foo handle_foo(); ++num_foos_seen; 4759@end verbatim 4760@end example 4761 4762is (rather surprisingly) truncated to 4763 4764@example 4765@verbatim 4766 foo handle_foo(); 4767@end verbatim 4768@end example 4769 4770@code{flex} does not truncate the action. Actions that are not enclosed 4771in braces are simply terminated at the end of the line. 4772 4773@node Memory Management, Serialized Tables, Lex and Posix, Top 4774@chapter Memory Management 4775 4776@cindex memory management 4777@anchor{memory-management} 4778This chapter describes how flex handles dynamic memory, and how you can 4779override the default behavior. 4780 4781@menu 4782* The Default Memory Management:: 4783* Overriding The Default Memory Management:: 4784* A Note About yytext And Memory:: 4785@end menu 4786 4787@node The Default Memory Management, Overriding The Default Memory Management, Memory Management, Memory Management 4788@section The Default Memory Management 4789 4790Flex allocates dynamic memory during initialization, and once in a while from 4791within a call to yylex(). Initialization takes place during the first call to 4792yylex(). Thereafter, flex may reallocate more memory if it needs to enlarge a 4793buffer. As of version 2.5.9 Flex will clean up all memory when you call @code{yylex_destroy} 4794@xref{faq-memory-leak}. 4795 4796Flex allocates dynamic memory for four purposes, listed below @footnote{The 4797quantities given here are approximate, and may vary due to host architecture, 4798compiler configuration, or due to future enhancements to flex.} 4799 4800@table @asis 4801 4802@item 16kB for the input buffer. 4803Flex allocates memory for the character buffer used to perform pattern 4804matching. Flex must read ahead from the input stream and store it in a large 4805character buffer. This buffer is typically the largest chunk of dynamic memory 4806flex consumes. This buffer will grow if necessary, doubling the size each time. 4807Flex frees this memory when you call yylex_destroy(). The default size of this 4808buffer (16384 bytes) is almost always too large. The ideal size for this 4809buffer is the length of the longest token expected, in bytes, plus a little more. Flex will allocate a few 4810extra bytes for housekeeping. Currently, to override the size of the input buffer 4811you must @code{#define YY_BUF_SIZE} to whatever number of bytes you want. We don't plan 4812to change this in the near future, but we reserve the right to do so if we ever add a more robust memory management 4813API. 4814 4815@item 64kb for the REJECT state. This will only be allocated if you use REJECT. 4816The size is large enough to hold the same number of states as characters in the input buffer. If you override the size of the 4817input buffer (via @code{YY_BUF_SIZE}), then you automatically override the size of this buffer as well. 4818 4819@item 100 bytes for the start condition stack. 4820Flex allocates memory for the start condition stack. This is the stack used 4821for pushing start states, i.e., with yy_push_state(). It will grow if 4822necessary. Since the states are simply integers, this stack doesn't consume 4823much memory. This stack is not present if @code{%option stack} is not 4824specified. You will rarely need to tune this buffer. The ideal size for this 4825stack is the maximum depth expected. The memory for this stack is 4826automatically destroyed when you call yylex_destroy(). @xref{option-stack}. 4827 4828@item 40 bytes for each YY_BUFFER_STATE. 4829Flex allocates memory for each YY_BUFFER_STATE. The buffer state itself 4830is about 40 bytes, plus an additional large character buffer (described above.) 4831The initial buffer state is created during initialization, and with each call 4832to yy_create_buffer(). You can't tune the size of this, but you can tune the 4833character buffer as described above. Any buffer state that you explicitly 4834create by calling yy_create_buffer() is @emph{NOT} destroyed automatically. You 4835must call yy_delete_buffer() to free the memory. The exception to this rule is 4836that flex will delete the current buffer automatically when you call 4837yylex_destroy(). If you delete the current buffer, be sure to set it to NULL. 4838That way, flex will not try to delete the buffer a second time (possibly 4839crashing your program!) At the time of this writing, flex does not provide a 4840growable stack for the buffer states. You have to manage that yourself. 4841@xref{Multiple Input Buffers}. 4842 4843@item 84 bytes for the reentrant scanner guts 4844Flex allocates about 84 bytes for the reentrant scanner structure when 4845you call yylex_init(). It is destroyed when the user calls yylex_destroy(). 4846 4847@end table 4848 4849 4850@node Overriding The Default Memory Management, A Note About yytext And Memory, The Default Memory Management, Memory Management 4851@section Overriding The Default Memory Management 4852 4853@cindex yyalloc, overriding 4854@cindex yyrealloc, overriding 4855@cindex yyfree, overriding 4856 4857Flex calls the functions @code{yyalloc}, @code{yyrealloc}, and @code{yyfree} 4858when it needs to allocate or free memory. By default, these functions are 4859wrappers around the standard C functions, @code{malloc}, @code{realloc}, and 4860@code{free}, respectively. You can override the default implementations by telling 4861flex that you will provide your own implementations. 4862 4863To override the default implementations, you must do two things: 4864 4865@enumerate 4866 4867@item Suppress the default implementations by specifying one or more of the 4868following options: 4869 4870@itemize 4871@opindex noyyalloc 4872@item @code{%option noyyalloc} 4873@item @code{%option noyyrealloc} 4874@item @code{%option noyyfree}. 4875@end itemize 4876 4877@item Provide your own implementation of the following functions: @footnote{It 4878is not necessary to override all (or any) of the memory management routines. 4879You may, for example, override @code{yyrealloc}, but not @code{yyfree} or 4880@code{yyalloc}.} 4881 4882@example 4883@verbatim 4884// For a non-reentrant scanner 4885void * yyalloc (size_t bytes); 4886void * yyrealloc (void * ptr, size_t bytes); 4887void yyfree (void * ptr); 4888 4889// For a reentrant scanner 4890void * yyalloc (size_t bytes, void * yyscanner); 4891void * yyrealloc (void * ptr, size_t bytes, void * yyscanner); 4892void yyfree (void * ptr, void * yyscanner); 4893@end verbatim 4894@end example 4895 4896@end enumerate 4897 4898In the following example, we will override all three memory routines. We assume 4899that there is a custom allocator with garbage collection. In order to make this 4900example interesting, we will use a reentrant scanner, passing a pointer to the 4901custom allocator through @code{yyextra}. 4902 4903@cindex overriding the memory routines 4904@example 4905@verbatim 4906%{ 4907#include "some_allocator.h" 4908%} 4909 4910/* Suppress the default implementations. */ 4911%option noyyalloc noyyrealloc noyyfree 4912%option reentrant 4913 4914/* Initialize the allocator. */ 4915#define YY_EXTRA_TYPE struct allocator* 4916#define YY_USER_INIT yyextra = allocator_create(); 4917 4918%% 4919.|\n ; 4920%% 4921 4922/* Provide our own implementations. */ 4923void * yyalloc (size_t bytes, void* yyscanner) { 4924 return allocator_alloc (yyextra, bytes); 4925} 4926 4927void * yyrealloc (void * ptr, size_t bytes, void* yyscanner) { 4928 return allocator_realloc (yyextra, bytes); 4929} 4930 4931void yyfree (void * ptr, void * yyscanner) { 4932 /* Do nothing -- we leave it to the garbage collector. */ 4933} 4934 4935@end verbatim 4936@end example 4937 4938 4939@node A Note About yytext And Memory, , Overriding The Default Memory Management, Memory Management 4940@section A Note About yytext And Memory 4941 4942@cindex yytext, memory considerations 4943 4944When flex finds a match, @code{yytext} points to the first character of the 4945match in the input buffer. The string itself is part of the input buffer, and 4946is @emph{NOT} allocated separately. The value of yytext will be overwritten the next 4947time yylex() is called. In short, the value of yytext is only valid from within 4948the matched rule's action. 4949 4950Often, you want the value of yytext to persist for later processing, i.e., by a 4951parser with non-zero lookahead. In order to preserve yytext, you will have to 4952copy it with strdup() or a similar function. But this introduces some headache 4953because your parser is now responsible for freeing the copy of yytext. If you 4954use a yacc or bison parser, (commonly used with flex), you will discover that 4955the error recovery mechanisms can cause memory to be leaked. 4956 4957To prevent memory leaks from strdup'd yytext, you will have to track the memory 4958somehow. Our experience has shown that a garbage collection mechanism or a 4959pooled memory mechanism will save you a lot of grief when writing parsers. 4960 4961@node Serialized Tables, Diagnostics, Memory Management, Top 4962@chapter Serialized Tables 4963@cindex serialization 4964@cindex memory, serialized tables 4965 4966@anchor{serialization} 4967A @code{flex} scanner has the ability to save the DFA tables to a file, and 4968load them at runtime when needed. The motivation for this feature is to reduce 4969the runtime memory footprint. Traditionally, these tables have been compiled into 4970the scanner as C arrays, and are sometimes quite large. Since the tables are 4971compiled into the scanner, the memory used by the tables can never be freed. 4972This is a waste of memory, especially if an application uses several scanners, 4973but none of them at the same time. 4974 4975The serialization feature allows the tables to be loaded at runtime, before 4976scanning begins. The tables may be discarded when scanning is finished. 4977 4978@menu 4979* Creating Serialized Tables:: 4980* Loading and Unloading Serialized Tables:: 4981* Tables File Format:: 4982@end menu 4983 4984@node Creating Serialized Tables, Loading and Unloading Serialized Tables, Serialized Tables, Serialized Tables 4985@section Creating Serialized Tables 4986@cindex tables, creating serialized 4987@cindex serialization of tables 4988 4989You may create a scanner with serialized tables by specifying: 4990 4991@example 4992@verbatim 4993 %option tables-file=FILE 4994or 4995 --tables-file=FILE 4996@end verbatim 4997@end example 4998 4999These options instruct flex to save the DFA tables to the file @var{FILE}. The tables 5000will @emph{not} be embedded in the generated scanner. The scanner will not 5001function on its own. The scanner will be dependent upon the serialized tables. You must 5002load the tables from this file at runtime before you can scan anything. 5003 5004If you do not specify a filename to @code{--tables-file}, the tables will be 5005saved to @file{lex.yy.tables}, where @samp{yy} is the appropriate prefix. 5006 5007If your project uses several different scanners, you can concatenate the 5008serialized tables into one file, and flex will find the correct set of tables, 5009using the scanner prefix as part of the lookup key. An example follows: 5010 5011@cindex serialized tables, multiple scanners 5012@example 5013@verbatim 5014$ flex --tables-file --prefix=cpp cpp.l 5015$ flex --tables-file --prefix=c c.l 5016$ cat lex.cpp.tables lex.c.tables > all.tables 5017@end verbatim 5018@end example 5019 5020The above example created two scanners, @samp{cpp}, and @samp{c}. Since we did 5021not specify a filename, the tables were serialized to @file{lex.c.tables} and 5022@file{lex.cpp.tables}, respectively. Then, we concatenated the two files 5023together into @file{all.tables}, which we will distribute with our project. At 5024runtime, we will open the file and tell flex to load the tables from it. Flex 5025will find the correct tables automatically. (See next section). 5026 5027@node Loading and Unloading Serialized Tables, Tables File Format, Creating Serialized Tables, Serialized Tables 5028@section Loading and Unloading Serialized Tables 5029@cindex tables, loading and unloading 5030@cindex loading tables at runtime 5031@cindex tables, freeing 5032@cindex freeing tables 5033@cindex memory, serialized tables 5034 5035If you've built your scanner with @code{%option tables-file}, then you must 5036load the scanner tables at runtime. This can be accomplished with the following 5037function: 5038 5039@deftypefun int yytables_fload (FILE* @var{fp} [, yyscan_t @var{scanner}]) 5040Locates scanner tables in the stream pointed to by @var{fp} and loads them. 5041Memory for the tables is allocated via @code{yyalloc}. You must call this 5042function before the first call to @code{yylex}. The argument @var{scanner} 5043only appears in the reentrant scanner. 5044This function returns @samp{0} (zero) on success, or non-zero on error. 5045@end deftypefun 5046 5047The loaded tables are @strong{not} automatically destroyed (unloaded) when you 5048call @code{yylex_destroy}. The reason is that you may create several scanners 5049of the same type (in a reentrant scanner), each of which needs access to these 5050tables. To avoid a nasty memory leak, you must call the following function: 5051 5052@deftypefun int yytables_destroy ([yyscan_t @var{scanner}]) 5053Unloads the scanner tables. The tables must be loaded again before you can scan 5054any more data. The argument @var{scanner} only appears in the reentrant 5055scanner. This function returns @samp{0} (zero) on success, or non-zero on 5056error. 5057@end deftypefun 5058 5059@strong{The functions @code{yytables_fload} and @code{yytables_destroy} are not 5060thread-safe.} You must ensure that these functions are called exactly once (for 5061each scanner type) in a threaded program, before any thread calls @code{yylex}. 5062After the tables are loaded, they are never written to, and no thread 5063protection is required thereafter -- until you destroy them. 5064 5065@node Tables File Format, , Loading and Unloading Serialized Tables, Serialized Tables 5066@section Tables File Format 5067@cindex tables, file format 5068@cindex file format, serialized tables 5069 5070This section defines the file format of serialized @code{flex} tables. 5071 5072The tables format allows for one or more sets of tables to be 5073specified, where each set corresponds to a given scanner. Scanners are 5074indexed by name, as described below. The file format is as follows: 5075 5076@example 5077@verbatim 5078 TABLE SET 1 5079 +-------------------------------+ 5080 Header | uint32 th_magic; | 5081 | uint32 th_hsize; | 5082 | uint32 th_ssize; | 5083 | uint16 th_flags; | 5084 | char th_version[]; | 5085 | char th_name[]; | 5086 | uint8 th_pad64[]; | 5087 +-------------------------------+ 5088 Table 1 | uint16 td_id; | 5089 | uint16 td_flags; | 5090 | uint32 td_hilen; | 5091 | uint32 td_lolen; | 5092 | void td_data[]; | 5093 | uint8 td_pad64[]; | 5094 +-------------------------------+ 5095 Table 2 | | 5096 . . . 5097 . . . 5098 . . . 5099 . . . 5100 Table n | | 5101 +-------------------------------+ 5102 TABLE SET 2 5103 . 5104 . 5105 . 5106 TABLE SET N 5107@end verbatim 5108@end example 5109 5110The above diagram shows that a complete set of tables consists of a header 5111followed by multiple individual tables. Furthermore, multiple complete sets may 5112be present in the same file, each set with its own header and tables. The sets 5113are contiguous in the file. The only way to know if another set follows is to 5114check the next four bytes for the magic number (or check for EOF). The header 5115and tables sections are padded to 64-bit boundaries. Below we describe each 5116field in detail. This format does not specify how the scanner will expand the 5117given data, i.e., data may be serialized as int8, but expanded to an int32 5118array at runtime. This is to reduce the size of the serialized data where 5119possible. Remember, @emph{all integer values are in network byte order}. 5120 5121@noindent 5122Fields of a table header: 5123 5124@table @code 5125@item th_magic 5126Magic number, always 0xF13C57B1. 5127 5128@item th_hsize 5129Size of this entire header, in bytes, including all fields plus any padding. 5130 5131@item th_ssize 5132Size of this entire set, in bytes, including the header, all tables, plus 5133any padding. 5134 5135@item th_flags 5136Bit flags for this table set. Currently unused. 5137 5138@item th_version[] 5139Flex version in NULL-terminated string format. e.g., @samp{2.5.13a}. This is 5140the version of flex that was used to create the serialized tables. 5141 5142@item th_name[] 5143Contains the name of this table set. The default is @samp{yytables}, 5144and is prefixed accordingly, e.g., @samp{footables}. Must be NULL-terminated. 5145 5146@item th_pad64[] 5147Zero or more NULL bytes, padding the entire header to the next 64-bit boundary 5148as calculated from the beginning of the header. 5149@end table 5150 5151@noindent 5152Fields of a table: 5153 5154@table @code 5155@item td_id 5156Specifies the table identifier. Possible values are: 5157@table @code 5158@item YYTD_ID_ACCEPT (0x01) 5159@code{yy_accept} 5160@item YYTD_ID_BASE (0x02) 5161@code{yy_base} 5162@item YYTD_ID_CHK (0x03) 5163@code{yy_chk} 5164@item YYTD_ID_DEF (0x04) 5165@code{yy_def} 5166@item YYTD_ID_EC (0x05) 5167@code{yy_ec } 5168@item YYTD_ID_META (0x06) 5169@code{yy_meta} 5170@item YYTD_ID_NUL_TRANS (0x07) 5171@code{yy_NUL_trans} 5172@item YYTD_ID_NXT (0x08) 5173@code{yy_nxt}. This array may be two dimensional. See the @code{td_hilen} 5174field below. 5175@item YYTD_ID_RULE_CAN_MATCH_EOL (0x09) 5176@code{yy_rule_can_match_eol} 5177@item YYTD_ID_START_STATE_LIST (0x0A) 5178@code{yy_start_state_list}. This array is handled specially because it is an 5179array of pointers to structs. See the @code{td_flags} field below. 5180@item YYTD_ID_TRANSITION (0x0B) 5181@code{yy_transition}. This array is handled specially because it is an array of 5182structs. See the @code{td_lolen} field below. 5183@item YYTD_ID_ACCLIST (0x0C) 5184@code{yy_acclist} 5185@end table 5186 5187@item td_flags 5188Bit flags describing how to interpret the data in @code{td_data}. 5189The data arrays are one-dimensional by default, but may be 5190two dimensional as specified in the @code{td_hilen} field. 5191 5192@table @code 5193@item YYTD_DATA8 (0x01) 5194The data is serialized as an array of type int8. 5195@item YYTD_DATA16 (0x02) 5196The data is serialized as an array of type int16. 5197@item YYTD_DATA32 (0x04) 5198The data is serialized as an array of type int32. 5199@item YYTD_PTRANS (0x08) 5200The data is a list of indexes of entries in the expanded @code{yy_transition} 5201array. Each index should be expanded to a pointer to the corresponding entry 5202in the @code{yy_transition} array. We count on the fact that the 5203@code{yy_transition} array has already been seen. 5204@item YYTD_STRUCT (0x10) 5205The data is a list of yy_trans_info structs, each of which consists of 5206two integers. There is no padding between struct elements or between structs. 5207The type of each member is determined by the @code{YYTD_DATA*} bits. 5208@end table 5209 5210@item td_hilen 5211If @code{td_hilen} is non-zero, then the data is a two-dimensional array. 5212Otherwise, the data is a one-dimensional array. @code{td_hilen} contains the 5213number of elements in the higher dimensional array, and @code{td_lolen} contains 5214the number of elements in the lowest dimension. 5215 5216Conceptually, @code{td_data} is either @code{sometype td_data[td_lolen]}, or 5217@code{sometype td_data[td_hilen][td_lolen]}, where @code{sometype} is specified 5218by the @code{td_flags} field. It is possible for both @code{td_lolen} and 5219@code{td_hilen} to be zero, in which case @code{td_data} is a zero length 5220array, and no data is loaded, i.e., this table is simply skipped. Flex does not 5221currently generate tables of zero length. 5222 5223@item td_lolen 5224Specifies the number of elements in the lowest dimension array. If this is 5225a one-dimensional array, then it is simply the number of elements in this array. 5226The element size is determined by the @code{td_flags} field. 5227 5228@item td_data[] 5229The table data. This array may be a one- or two-dimensional array, of type 5230@code{int8}, @code{int16}, @code{int32}, @code{struct yy_trans_info}, or 5231@code{struct yy_trans_info*}, depending upon the values in the 5232@code{td_flags}, @code{td_hilen}, and @code{td_lolen} fields. 5233 5234@item td_pad64[] 5235Zero or more NULL bytes, padding the entire table to the next 64-bit boundary as 5236calculated from the beginning of this table. 5237@end table 5238 5239@node Diagnostics, Limitations, Serialized Tables, Top 5240@chapter Diagnostics 5241 5242@cindex error reporting, diagnostic messages 5243@cindex warnings, diagnostic messages 5244 5245The following is a list of @code{flex} diagnostic messages: 5246 5247@itemize 5248@item 5249@samp{warning, rule cannot be matched} indicates that the given rule 5250cannot be matched because it follows other rules that will always match 5251the same text as it. For example, in the following @samp{foo} cannot be 5252matched because it comes after an identifier ``catch-all'' rule: 5253 5254@cindex warning, rule cannot be matched 5255@example 5256@verbatim 5257 [a-z]+ got_identifier(); 5258 foo got_foo(); 5259@end verbatim 5260@end example 5261 5262Using @code{REJECT} in a scanner suppresses this warning. 5263 5264@item 5265@samp{warning, -s option given but default rule can be matched} means 5266that it is possible (perhaps only in a particular start condition) that 5267the default rule (match any single character) is the only one that will 5268match a particular input. Since @samp{-s} was given, presumably this is 5269not intended. 5270 5271@item 5272@code{reject_used_but_not_detected undefined} or 5273@code{yymore_used_but_not_detected undefined}. These errors can occur 5274at compile time. They indicate that the scanner uses @code{REJECT} or 5275@code{yymore()} but that @code{flex} failed to notice the fact, meaning 5276that @code{flex} scanned the first two sections looking for occurrences 5277of these actions and failed to find any, but somehow you snuck some in 5278(via a #include file, for example). Use @code{%option reject} or 5279@code{%option yymore} to indicate to @code{flex} that you really do use 5280these features. 5281 5282@item 5283@samp{flex scanner jammed}. a scanner compiled with 5284@samp{-s} has encountered an input string which wasn't matched by any of 5285its rules. This error can also occur due to internal problems. 5286 5287@item 5288@samp{token too large, exceeds YYLMAX}. your scanner uses @code{%array} 5289and one of its rules matched a string longer than the @code{YYLMAX} 5290constant (8K bytes by default). You can increase the value by 5291#define'ing @code{YYLMAX} in the definitions section of your @code{flex} 5292input. 5293 5294@item 5295@samp{scanner requires -8 flag to use the character 'x'}. Your scanner 5296specification includes recognizing the 8-bit character @samp{'x'} and 5297you did not specify the -8 flag, and your scanner defaulted to 7-bit 5298because you used the @samp{-Cf} or @samp{-CF} table compression options. 5299See the discussion of the @samp{-7} flag, @ref{Scanner Options}, for 5300details. 5301 5302@item 5303@samp{flex scanner push-back overflow}. you used @code{unput()} to push 5304back so much text that the scanner's buffer could not hold both the 5305pushed-back text and the current token in @code{yytext}. Ideally the 5306scanner should dynamically resize the buffer in this case, but at 5307present it does not. 5308 5309@item 5310@samp{input buffer overflow, can't enlarge buffer because scanner uses 5311REJECT}. the scanner was working on matching an extremely large token 5312and needed to expand the input buffer. This doesn't work with scanners 5313that use @code{REJECT}. 5314 5315@item 5316@samp{fatal flex scanner internal error--end of buffer missed}. This can 5317occur in a scanner which is reentered after a long-jump has jumped out 5318(or over) the scanner's activation frame. Before reentering the 5319scanner, use: 5320@example 5321@verbatim 5322 yyrestart( yyin ); 5323@end verbatim 5324@end example 5325or, as noted above, switch to using the C++ scanner class. 5326 5327@item 5328@samp{too many start conditions in <> construct!} you listed more start 5329conditions in a <> construct than exist (so you must have listed at 5330least one of them twice). 5331@end itemize 5332 5333@node Limitations, Bibliography, Diagnostics, Top 5334@chapter Limitations 5335 5336@cindex limitations of flex 5337 5338Some trailing context patterns cannot be properly matched and generate 5339warning messages (@samp{dangerous trailing context}). These are 5340patterns where the ending of the first part of the rule matches the 5341beginning of the second part, such as @samp{zx*/xy*}, where the 'x*' 5342matches the 'x' at the beginning of the trailing context. (Note that 5343the POSIX draft states that the text matched by such patterns is 5344undefined.) For some trailing context rules, parts which are actually 5345fixed-length are not recognized as such, leading to the abovementioned 5346performance loss. In particular, parts using @samp{|} or @samp{@{n@}} 5347(such as @samp{foo@{3@}}) are always considered variable-length. 5348Combining trailing context with the special @samp{|} action can result 5349in @emph{fixed} trailing context being turned into the more expensive 5350@emph{variable} trailing context. For example, in the following: 5351 5352@cindex warning, dangerous trailing context 5353@example 5354@verbatim 5355 %% 5356 abc | 5357 xyz/def 5358@end verbatim 5359@end example 5360 5361Use of @code{unput()} invalidates yytext and yyleng, unless the 5362@code{%array} directive or the @samp{-l} option has been used. 5363Pattern-matching of @code{NUL}s is substantially slower than matching 5364other characters. Dynamic resizing of the input buffer is slow, as it 5365entails rescanning all the text matched so far by the current (generally 5366huge) token. Due to both buffering of input and read-ahead, you cannot 5367intermix calls to @file{<stdio.h>} routines, such as, @b{getchar()}, 5368with @code{flex} rules and expect it to work. Call @code{input()} 5369instead. The total table entries listed by the @samp{-v} flag excludes 5370the number of table entries needed to determine what rule has been 5371matched. The number of entries is equal to the number of DFA states if 5372the scanner does not use @code{REJECT}, and somewhat greater than the 5373number of states if it does. @code{REJECT} cannot be used with the 5374@samp{-f} or @samp{-F} options. 5375 5376The @code{flex} internal algorithms need documentation. 5377 5378@node Bibliography, FAQ, Limitations, Top 5379@chapter Additional Reading 5380 5381You may wish to read more about the following programs: 5382@itemize 5383@item lex 5384@item yacc 5385@item sed 5386@item awk 5387@end itemize 5388 5389The following books may contain material of interest: 5390 5391John Levine, Tony Mason, and Doug Brown, 5392@emph{Lex & Yacc}, 5393O'Reilly and Associates. Be sure to get the 2nd edition. 5394 5395M. E. Lesk and E. Schmidt, 5396@emph{LEX -- Lexical Analyzer Generator} 5397 5398Alfred Aho, Ravi Sethi and Jeffrey Ullman, @emph{Compilers: Principles, 5399Techniques and Tools}, Addison-Wesley (1986). Describes the 5400pattern-matching techniques used by @code{flex} (deterministic finite 5401automata). 5402 5403@node FAQ, Appendices, Bibliography, Top 5404@unnumbered FAQ 5405 5406From time to time, the @code{flex} maintainer receives certain 5407questions. Rather than repeat answers to well-understood problems, we 5408publish them here. 5409 5410@menu 5411* When was flex born?:: 5412* How do I expand backslash-escape sequences in C-style quoted strings?:: 5413* Why do flex scanners call fileno if it is not ANSI compatible?:: 5414* Does flex support recursive pattern definitions?:: 5415* How do I skip huge chunks of input (tens of megabytes) while using flex?:: 5416* Flex is not matching my patterns in the same order that I defined them.:: 5417* My actions are executing out of order or sometimes not at all.:: 5418* How can I have multiple input sources feed into the same scanner at the same time?:: 5419* Can I build nested parsers that work with the same input file?:: 5420* How can I match text only at the end of a file?:: 5421* How can I make REJECT cascade across start condition boundaries?:: 5422* Why cant I use fast or full tables with interactive mode?:: 5423* How much faster is -F or -f than -C?:: 5424* If I have a simple grammar cant I just parse it with flex?:: 5425* Why doesn't yyrestart() set the start state back to INITIAL?:: 5426* How can I match C-style comments?:: 5427* The period isn't working the way I expected.:: 5428* Can I get the flex manual in another format?:: 5429* Does there exist a "faster" NDFA->DFA algorithm?:: 5430* How does flex compile the DFA so quickly?:: 5431* How can I use more than 8192 rules?:: 5432* How do I abandon a file in the middle of a scan and switch to a new file?:: 5433* How do I execute code only during initialization (only before the first scan)?:: 5434* How do I execute code at termination?:: 5435* Where else can I find help?:: 5436* Can I include comments in the "rules" section of the file?:: 5437* I get an error about undefined yywrap().:: 5438* How can I change the matching pattern at run time?:: 5439* How can I expand macros in the input?:: 5440* How can I build a two-pass scanner?:: 5441* How do I match any string not matched in the preceding rules?:: 5442* I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: 5443* Is there a way to make flex treat NULL like a regular character?:: 5444* Whenever flex can not match the input it says "flex scanner jammed".:: 5445* Why doesn't flex have non-greedy operators like perl does?:: 5446* Memory leak - 16386 bytes allocated by malloc.:: 5447* How do I track the byte offset for lseek()?:: 5448* How do I use my own I/O classes in a C++ scanner?:: 5449* How do I skip as many chars as possible?:: 5450* deleteme00:: 5451* Are certain equivalent patterns faster than others?:: 5452* Is backing up a big deal?:: 5453* Can I fake multi-byte character support?:: 5454* deleteme01:: 5455* Can you discuss some flex internals?:: 5456* unput() messes up yy_at_bol:: 5457* The | operator is not doing what I want:: 5458* Why can't flex understand this variable trailing context pattern?:: 5459* The ^ operator isn't working:: 5460* Trailing context is getting confused with trailing optional patterns:: 5461* Is flex GNU or not?:: 5462* ERASEME53:: 5463* I need to scan if-then-else blocks and while loops:: 5464* ERASEME55:: 5465* ERASEME56:: 5466* ERASEME57:: 5467* Is there a repository for flex scanners?:: 5468* How can I conditionally compile or preprocess my flex input file?:: 5469* Where can I find grammars for lex and yacc?:: 5470* I get an end-of-buffer message for each character scanned.:: 5471* unnamed-faq-62:: 5472* unnamed-faq-63:: 5473* unnamed-faq-64:: 5474* unnamed-faq-65:: 5475* unnamed-faq-66:: 5476* unnamed-faq-67:: 5477* unnamed-faq-68:: 5478* unnamed-faq-69:: 5479* unnamed-faq-70:: 5480* unnamed-faq-71:: 5481* unnamed-faq-72:: 5482* unnamed-faq-73:: 5483* unnamed-faq-74:: 5484* unnamed-faq-75:: 5485* unnamed-faq-76:: 5486* unnamed-faq-77:: 5487* unnamed-faq-78:: 5488* unnamed-faq-79:: 5489* unnamed-faq-80:: 5490* unnamed-faq-81:: 5491* unnamed-faq-82:: 5492* unnamed-faq-83:: 5493* unnamed-faq-84:: 5494* unnamed-faq-85:: 5495* unnamed-faq-86:: 5496* unnamed-faq-87:: 5497* unnamed-faq-88:: 5498* unnamed-faq-90:: 5499* unnamed-faq-91:: 5500* unnamed-faq-92:: 5501* unnamed-faq-93:: 5502* unnamed-faq-94:: 5503* unnamed-faq-95:: 5504* unnamed-faq-96:: 5505* unnamed-faq-97:: 5506* unnamed-faq-98:: 5507* unnamed-faq-99:: 5508* unnamed-faq-100:: 5509* unnamed-faq-101:: 5510* What is the difference between YYLEX_PARAM and YY_DECL?:: 5511* Why do I get "conflicting types for yylex" error?:: 5512* How do I access the values set in a Flex action from within a Bison action?:: 5513@end menu 5514 5515@node When was flex born? 5516@unnumberedsec When was flex born? 5517 5518Vern Paxson took over 5519the @cite{Software Tools} lex project from Jef Poskanzer in 1982. At that point it 5520was written in Ratfor. Around 1987 or so, Paxson translated it into C, and 5521a legend was born :-). 5522 5523@node How do I expand backslash-escape sequences in C-style quoted strings? 5524@unnumberedsec How do I expand backslash-escape sequences in C-style quoted strings? 5525 5526A key point when scanning quoted strings is that you cannot (easily) write 5527a single rule that will precisely match the string if you allow things 5528like embedded escape sequences and newlines. If you try to match strings 5529with a single rule then you'll wind up having to rescan the string anyway 5530to find any escape sequences. 5531 5532Instead you can use exclusive start conditions and a set of rules, one for 5533matching non-escaped text, one for matching a single escape, one for 5534matching an embedded newline, and one for recognizing the end of the 5535string. Each of these rules is then faced with the question of where to 5536put its intermediary results. The best solution is for the rules to 5537append their local value of @code{yytext} to the end of a ``string literal'' 5538buffer. A rule like the escape-matcher will append to the buffer the 5539meaning of the escape sequence rather than the literal text in @code{yytext}. 5540In this way, @code{yytext} does not need to be modified at all. 5541 5542@node Why do flex scanners call fileno if it is not ANSI compatible? 5543@unnumberedsec Why do flex scanners call fileno if it is not ANSI compatible? 5544 5545Flex scanners call @code{fileno()} in order to get the file descriptor 5546corresponding to @code{yyin}. The file descriptor may be passed to 5547@code{isatty()} or @code{read()}, depending upon which @code{%options} you specified. 5548If your system does not have @code{fileno()} support, to get rid of the 5549@code{read()} call, do not specify @code{%option read}. To get rid of the @code{isatty()} 5550call, you must specify one of @code{%option always-interactive} or 5551@code{%option never-interactive}. 5552 5553@node Does flex support recursive pattern definitions? 5554@unnumberedsec Does flex support recursive pattern definitions? 5555 5556e.g., 5557 5558@example 5559@verbatim 5560%% 5561block "{"({block}|{statement})*"}" 5562@end verbatim 5563@end example 5564 5565No. You cannot have recursive definitions. The pattern-matching power of 5566regular expressions in general (and therefore flex scanners, too) is 5567limited. In particular, regular expressions cannot ``balance'' parentheses 5568to an arbitrary degree. For example, it's impossible to write a regular 5569expression that matches all strings containing the same number of '@{'s 5570as '@}'s. For more powerful pattern matching, you need a parser, such 5571as @cite{GNU bison}. 5572 5573@node How do I skip huge chunks of input (tens of megabytes) while using flex? 5574@unnumberedsec How do I skip huge chunks of input (tens of megabytes) while using flex? 5575 5576Use @code{fseek()} (or @code{lseek()}) to position yyin, then call @code{yyrestart()}. 5577 5578@node Flex is not matching my patterns in the same order that I defined them. 5579@unnumberedsec Flex is not matching my patterns in the same order that I defined them. 5580 5581@code{flex} picks the 5582rule that matches the most text (i.e., the longest possible input string). 5583This is because @code{flex} uses an entirely different matching technique 5584(``deterministic finite automata'') that actually does all of the matching 5585simultaneously, in parallel. (Seems impossible, but it's actually a fairly 5586simple technique once you understand the principles.) 5587 5588A side-effect of this parallel matching is that when the input matches more 5589than one rule, @code{flex} scanners pick the rule that matched the @emph{most} text. This 5590is explained further in the manual, in the section @xref{Matching}. 5591 5592If you want @code{flex} to choose a shorter match, then you can work around this 5593behavior by expanding your short 5594rule to match more text, then put back the extra: 5595 5596@example 5597@verbatim 5598data_.* yyless( 5 ); BEGIN BLOCKIDSTATE; 5599@end verbatim 5600@end example 5601 5602Another fix would be to make the second rule active only during the 5603@code{<BLOCKIDSTATE>} start condition, and make that start condition exclusive 5604by declaring it with @code{%x} instead of @code{%s}. 5605 5606A final fix is to change the input language so that the ambiguity for 5607@samp{data_} is removed, by adding characters to it that don't match the 5608identifier rule, or by removing characters (such as @samp{_}) from the 5609identifier rule so it no longer matches @samp{data_}. (Of course, you might 5610also not have the option of changing the input language.) 5611 5612@node My actions are executing out of order or sometimes not at all. 5613@unnumberedsec My actions are executing out of order or sometimes not at all. 5614 5615Most likely, you have (in error) placed the opening @samp{@{} of the action 5616block on a different line than the rule, e.g., 5617 5618@example 5619@verbatim 5620^(foo|bar) 5621{ <<<--- WRONG! 5622 5623} 5624@end verbatim 5625@end example 5626 5627@code{flex} requires that the opening @samp{@{} of an action associated with a rule 5628begin on the same line as does the rule. You need instead to write your rules 5629as follows: 5630 5631@example 5632@verbatim 5633^(foo|bar) { // CORRECT! 5634 5635} 5636@end verbatim 5637@end example 5638 5639@node How can I have multiple input sources feed into the same scanner at the same time? 5640@unnumberedsec How can I have multiple input sources feed into the same scanner at the same time? 5641 5642If @dots{} 5643@itemize 5644@item 5645your scanner is free of backtracking (verified using @code{flex}'s @samp{-b} flag), 5646@item 5647AND you run your scanner interactively (@samp{-I} option; default unless using special table 5648compression options), 5649@item 5650AND you feed it one character at a time by redefining @code{YY_INPUT} to do so, 5651@end itemize 5652 5653then every time it matches a token, it will have exhausted its input 5654buffer (because the scanner is free of backtracking). This means you 5655can safely use @code{select()} at the point and only call @code{yylex()} for another 5656token if @code{select()} indicates there's data available. 5657 5658That is, move the @code{select()} out from the input function to a point where 5659it determines whether @code{yylex()} gets called for the next token. 5660 5661With this approach, you will still have problems if your input can arrive 5662piecemeal; @code{select()} could inform you that the beginning of a token is 5663available, you call @code{yylex()} to get it, but it winds up blocking waiting 5664for the later characters in the token. 5665 5666Here's another way: Move your input multiplexing inside of @code{YY_INPUT}. That 5667is, whenever @code{YY_INPUT} is called, it @code{select()}'s to see where input is 5668available. If input is available for the scanner, it reads and returns the 5669next byte. If input is available from another source, it calls whatever 5670function is responsible for reading from that source. (If no input is 5671available, it blocks until some input is available.) I've used this technique in an 5672interpreter I wrote that both reads keyboard input using a @code{flex} scanner and 5673IPC traffic from sockets, and it works fine. 5674 5675@node Can I build nested parsers that work with the same input file? 5676@unnumberedsec Can I build nested parsers that work with the same input file? 5677 5678This is not going to work without some additional effort. The reason is 5679that @code{flex} block-buffers the input it reads from @code{yyin}. This means that the 5680``outermost'' @code{yylex()}, when called, will automatically slurp up the first 8K 5681of input available on yyin, and subsequent calls to other @code{yylex()}'s won't 5682see that input. You might be tempted to work around this problem by 5683redefining @code{YY_INPUT} to only return a small amount of text, but it turns out 5684that that approach is quite difficult. Instead, the best solution is to 5685combine all of your scanners into one large scanner, using a different 5686exclusive start condition for each. 5687 5688@node How can I match text only at the end of a file? 5689@unnumberedsec How can I match text only at the end of a file? 5690 5691There is no way to write a rule which is ``match this text, but only if 5692it comes at the end of the file''. You can fake it, though, if you happen 5693to have a character lying around that you don't allow in your input. 5694Then you redefine @code{YY_INPUT} to call your own routine which, if it sees 5695an @samp{EOF}, returns the magic character first (and remembers to return a 5696real @code{EOF} next time it's called). Then you could write: 5697 5698@example 5699@verbatim 5700<COMMENT>(.|\n)*{EOF_CHAR} /* saw comment at EOF */ 5701@end verbatim 5702@end example 5703 5704@node How can I make REJECT cascade across start condition boundaries? 5705@unnumberedsec How can I make REJECT cascade across start condition boundaries? 5706 5707You can do this as follows. Suppose you have a start condition @samp{A}, and 5708after exhausting all of the possible matches in @samp{<A>}, you want to try 5709matches in @samp{<INITIAL>}. Then you could use the following: 5710 5711@example 5712@verbatim 5713%x A 5714%% 5715<A>rule_that_is_long ...; REJECT; 5716<A>rule ...; REJECT; /* shorter rule */ 5717<A>etc. 5718... 5719<A>.|\n { 5720/* Shortest and last rule in <A>, so 5721* cascaded REJECTs will eventually 5722* wind up matching this rule. We want 5723* to now switch to the initial state 5724* and try matching from there instead. 5725*/ 5726yyless(0); /* put back matched text */ 5727BEGIN(INITIAL); 5728} 5729@end verbatim 5730@end example 5731 5732@node Why cant I use fast or full tables with interactive mode? 5733@unnumberedsec Why can't I use fast or full tables with interactive mode? 5734 5735One of the assumptions 5736flex makes is that interactive applications are inherently slow (they're 5737waiting on a human after all). 5738It has to do with how the scanner detects that it must be finished scanning 5739a token. For interactive scanners, after scanning each character the current 5740state is looked up in a table (essentially) to see whether there's a chance 5741of another input character possibly extending the length of the match. If 5742not, the scanner halts. For non-interactive scanners, the end-of-token test 5743is much simpler, basically a compare with 0, so no memory bus cycles. Since 5744the test occurs in the innermost scanning loop, one would like to make it go 5745as fast as possible. 5746 5747Still, it seems reasonable to allow the user to choose to trade off a bit 5748of performance in this area to gain the corresponding flexibility. There 5749might be another reason, though, why fast scanners don't support the 5750interactive option. 5751 5752@node How much faster is -F or -f than -C? 5753@unnumberedsec How much faster is -F or -f than -C? 5754 5755Much faster (factor of 2-3). 5756 5757@node If I have a simple grammar cant I just parse it with flex? 5758@unnumberedsec If I have a simple grammar can't I just parse it with flex? 5759 5760Is your grammar recursive? That's almost always a sign that you're 5761better off using a parser/scanner rather than just trying to use a scanner 5762alone. 5763 5764@node Why doesn't yyrestart() set the start state back to INITIAL? 5765@unnumberedsec Why doesn't yyrestart() set the start state back to INITIAL? 5766 5767There are two reasons. The first is that there might 5768be programs that rely on the start state not changing across file changes. 5769The second is that beginning with @code{flex} version 2.4, use of @code{yyrestart()} is no longer required, 5770so fixing the problem there doesn't solve the more general problem. 5771 5772@node How can I match C-style comments? 5773@unnumberedsec How can I match C-style comments? 5774 5775You might be tempted to try something like this: 5776 5777@example 5778@verbatim 5779"/*".*"*/" // WRONG! 5780@end verbatim 5781@end example 5782 5783or, worse, this: 5784 5785@example 5786@verbatim 5787"/*"(.|\n)"*/" // WRONG! 5788@end verbatim 5789@end example 5790 5791The above rules will eat too much input, and blow up on things like: 5792 5793@example 5794@verbatim 5795/* a comment */ do_my_thing( "oops */" ); 5796@end verbatim 5797@end example 5798 5799Here is one way which allows you to track line information: 5800 5801@example 5802@verbatim 5803<INITIAL>{ 5804"/*" BEGIN(IN_COMMENT); 5805} 5806<IN_COMMENT>{ 5807"*/" BEGIN(INITIAL); 5808[^*\n]+ // eat comment in chunks 5809"*" // eat the lone star 5810\n yylineno++; 5811} 5812@end verbatim 5813@end example 5814 5815@node The period isn't working the way I expected. 5816@unnumberedsec The '.' isn't working the way I expected. 5817 5818Here are some tips for using @samp{.}: 5819 5820@itemize 5821@item 5822A common mistake is to place the grouping parenthesis AFTER an operator, when 5823you really meant to place the parenthesis BEFORE the operator, e.g., you 5824probably want this @code{(foo|bar)+} and NOT this @code{(foo|bar+)}. 5825 5826The first pattern matches the words @samp{foo} or @samp{bar} any number of 5827times, e.g., it matches the text @samp{barfoofoobarfoo}. The 5828second pattern matches a single instance of @code{foo} or a single instance of 5829@code{bar} followed by one or more @samp{r}s, e.g., it matches the text @code{barrrr} . 5830@item 5831A @samp{.} inside @samp{[]}'s just means a literal@samp{.} (period), 5832and NOT ``any character except newline''. 5833@item 5834Remember that @samp{.} matches any character EXCEPT @samp{\n} (and @samp{EOF}). 5835If you really want to match ANY character, including newlines, then use @code{(.|\n)} 5836Beware that the regex @code{(.|\n)+} will match your entire input! 5837@item 5838Finally, if you want to match a literal @samp{.} (a period), then use @samp{[.]} or @samp{"."} 5839@end itemize 5840 5841@node Can I get the flex manual in another format? 5842@unnumberedsec Can I get the flex manual in another format? 5843 5844The @code{flex} source distribution includes a texinfo manual. You are 5845free to convert that texinfo into whatever format you desire. The 5846@code{texinfo} package includes tools for conversion to a number of formats. 5847 5848@node Does there exist a "faster" NDFA->DFA algorithm? 5849@unnumberedsec Does there exist a "faster" NDFA->DFA algorithm? 5850 5851There's no way around the potential exponential running time - it 5852can take you exponential time just to enumerate all of the DFA states. 5853In practice, though, the running time is closer to linear, or sometimes 5854quadratic. 5855 5856@node How does flex compile the DFA so quickly? 5857@unnumberedsec How does flex compile the DFA so quickly? 5858 5859There are two big speed wins that @code{flex} uses: 5860 5861@enumerate 5862@item 5863It analyzes the input rules to construct equivalence classes for those 5864characters that always make the same transitions. It then rewrites the NFA 5865using equivalence classes for transitions instead of characters. This cuts 5866down the NFA->DFA computation time dramatically, to the point where, for 5867uncompressed DFA tables, the DFA generation is often I/O bound in writing out 5868the tables. 5869@item 5870It maintains hash values for previously computed DFA states, so testing 5871whether a newly constructed DFA state is equivalent to a previously constructed 5872state can be done very quickly, by first comparing hash values. 5873@end enumerate 5874 5875@node How can I use more than 8192 rules? 5876@unnumberedsec How can I use more than 8192 rules? 5877 5878@code{Flex} is compiled with an upper limit of 8192 rules per scanner. 5879If you need more than 8192 rules in your scanner, you'll have to recompile @code{flex} 5880with the following changes in @file{flexdef.h}: 5881 5882@example 5883@verbatim 5884< #define YY_TRAILING_MASK 0x2000 5885< #define YY_TRAILING_HEAD_MASK 0x4000 5886-- 5887> #define YY_TRAILING_MASK 0x20000000 5888> #define YY_TRAILING_HEAD_MASK 0x40000000 5889@end verbatim 5890@end example 5891 5892This should work okay as long as your C compiler uses 32 bit integers. 5893But you might want to think about whether using such a huge number of rules 5894is the best way to solve your problem. 5895 5896The following may also be relevant: 5897 5898With luck, you should be able to increase the definitions in flexdef.h for: 5899 5900@example 5901@verbatim 5902#define JAMSTATE -32766 /* marks a reference to the state that always jams */ 5903#define MAXIMUM_MNS 31999 5904#define BAD_SUBSCRIPT -32767 5905@end verbatim 5906@end example 5907 5908recompile everything, and it'll all work. Flex only has these 16-bit-like 5909values built into it because a long time ago it was developed on a machine 5910with 16-bit ints. I've given this advice to others in the past but haven't 5911heard back from them whether it worked okay or not... 5912 5913@node How do I abandon a file in the middle of a scan and switch to a new file? 5914@unnumberedsec How do I abandon a file in the middle of a scan and switch to a new file? 5915 5916Just call @code{yyrestart(newfile)}. Be sure to reset the start state if you want a 5917``fresh start, since @code{yyrestart} does NOT reset the start state back to @code{INITIAL}. 5918 5919@node How do I execute code only during initialization (only before the first scan)? 5920@unnumberedsec How do I execute code only during initialization (only before the first scan)? 5921 5922You can specify an initial action by defining the macro @code{YY_USER_INIT} (though 5923note that @code{yyout} may not be available at the time this macro is executed). Or you 5924can add to the beginning of your rules section: 5925 5926@example 5927@verbatim 5928%% 5929 /* Must be indented! */ 5930 static int did_init = 0; 5931 5932 if ( ! did_init ){ 5933do_my_init(); 5934 did_init = 1; 5935 } 5936@end verbatim 5937@end example 5938 5939@node How do I execute code at termination? 5940@unnumberedsec How do I execute code at termination? 5941 5942You can specify an action for the @code{<<EOF>>} rule. 5943 5944@node Where else can I find help? 5945@unnumberedsec Where else can I find help? 5946 5947You can find the flex homepage on the web at 5948@uref{http://flex.sourceforge.net/}. See that page for details about flex 5949mailing lists as well. 5950 5951@node Can I include comments in the "rules" section of the file? 5952@unnumberedsec Can I include comments in the "rules" section of the file? 5953 5954Yes, just about anywhere you want to. See the manual for the specific syntax. 5955 5956@node I get an error about undefined yywrap(). 5957@unnumberedsec I get an error about undefined yywrap(). 5958 5959You must supply a @code{yywrap()} function of your own, or link to @file{libfl.a} 5960(which provides one), or use 5961 5962@example 5963@verbatim 5964%option noyywrap 5965@end verbatim 5966@end example 5967 5968in your source to say you don't want a @code{yywrap()} function. 5969 5970@node How can I change the matching pattern at run time? 5971@unnumberedsec How can I change the matching pattern at run time? 5972 5973You can't, it's compiled into a static table when flex builds the scanner. 5974 5975@node How can I expand macros in the input? 5976@unnumberedsec How can I expand macros in the input? 5977 5978The best way to approach this problem is at a higher level, e.g., in the parser. 5979 5980However, you can do this using multiple input buffers. 5981 5982@example 5983@verbatim 5984%% 5985macro/[a-z]+ { 5986/* Saw the macro "macro" followed by extra stuff. */ 5987main_buffer = YY_CURRENT_BUFFER; 5988expansion_buffer = yy_scan_string(expand(yytext)); 5989yy_switch_to_buffer(expansion_buffer); 5990} 5991 5992<<EOF>> { 5993if ( expansion_buffer ) 5994{ 5995// We were doing an expansion, return to where 5996// we were. 5997yy_switch_to_buffer(main_buffer); 5998yy_delete_buffer(expansion_buffer); 5999expansion_buffer = 0; 6000} 6001else 6002yyterminate(); 6003} 6004@end verbatim 6005@end example 6006 6007You probably will want a stack of expansion buffers to allow nested macros. 6008From the above though hopefully the idea is clear. 6009 6010@node How can I build a two-pass scanner? 6011@unnumberedsec How can I build a two-pass scanner? 6012 6013One way to do it is to filter the first pass to a temporary file, 6014then process the temporary file on the second pass. You will probably see a 6015performance hit, due to all the disk I/O. 6016 6017When you need to look ahead far forward like this, it almost always means 6018that the right solution is to build a parse tree of the entire input, then 6019walk it after the parse in order to generate the output. In a sense, this 6020is a two-pass approach, once through the text and once through the parse 6021tree, but the performance hit for the latter is usually an order of magnitude 6022smaller, since everything is already classified, in binary format, and 6023residing in memory. 6024 6025@node How do I match any string not matched in the preceding rules? 6026@unnumberedsec How do I match any string not matched in the preceding rules? 6027 6028One way to assign precedence, is to place the more specific rules first. If 6029two rules would match the same input (same sequence of characters) then the 6030first rule listed in the @code{flex} input wins, e.g., 6031 6032@example 6033@verbatim 6034%% 6035foo[a-zA-Z_]+ return FOO_ID; 6036bar[a-zA-Z_]+ return BAR_ID; 6037[a-zA-Z_]+ return GENERIC_ID; 6038@end verbatim 6039@end example 6040 6041Note that the rule @code{[a-zA-Z_]+} must come *after* the others. It will match the 6042same amount of text as the more specific rules, and in that case the 6043@code{flex} scanner will pick the first rule listed in your scanner as the 6044one to match. 6045 6046@node I am trying to port code from AT&T lex that uses yysptr and yysbuf. 6047@unnumberedsec I am trying to port code from AT&T lex that uses yysptr and yysbuf. 6048 6049Those are internal variables pointing into the AT&T scanner's input buffer. I 6050imagine they're being manipulated in user versions of the @code{input()} and @code{unput()} 6051functions. If so, what you need to do is analyze those functions to figure out 6052what they're doing, and then replace @code{input()} with an appropriate definition of 6053@code{YY_INPUT}. You shouldn't need to (and must not) replace 6054@code{flex}'s @code{unput()} function. 6055 6056@node Is there a way to make flex treat NULL like a regular character? 6057@unnumberedsec Is there a way to make flex treat NULL like a regular character? 6058 6059Yes, @samp{\0} and @samp{\x00} should both do the trick. Perhaps you have an ancient 6060version of @code{flex}. The latest release is version @value{VERSION}. 6061 6062@node Whenever flex can not match the input it says "flex scanner jammed". 6063@unnumberedsec Whenever flex can not match the input it says "flex scanner jammed". 6064 6065You need to add a rule that matches the otherwise-unmatched text, 6066e.g., 6067 6068@example 6069@verbatim 6070%option yylineno 6071%% 6072[[a bunch of rules here]] 6073 6074. printf("bad input character '%s' at line %d\n", yytext, yylineno); 6075@end verbatim 6076@end example 6077 6078See @code{%option default} for more information. 6079 6080@node Why doesn't flex have non-greedy operators like perl does? 6081@unnumberedsec Why doesn't flex have non-greedy operators like perl does? 6082 6083A DFA can do a non-greedy match by stopping 6084the first time it enters an accepting state, instead of consuming input until 6085it determines that no further matching is possible (a ``jam'' state). This 6086is actually easier to implement than longest leftmost match (which flex does). 6087 6088But it's also much less useful than longest leftmost match. In general, 6089when you find yourself wishing for non-greedy matching, that's usually a 6090sign that you're trying to make the scanner do some parsing. That's 6091generally the wrong approach, since it lacks the power to do a decent job. 6092Better is to either introduce a separate parser, or to split the scanner 6093into multiple scanners using (exclusive) start conditions. 6094 6095You might have 6096a separate start state once you've seen the @samp{BEGIN}. In that state, you 6097might then have a regex that will match @samp{END} (to kick you out of the 6098state), and perhaps @samp{(.|\n)} to get a single character within the chunk ... 6099 6100This approach also has much better error-reporting properties. 6101 6102@node Memory leak - 16386 bytes allocated by malloc. 6103@unnumberedsec Memory leak - 16386 bytes allocated by malloc. 6104@anchor{faq-memory-leak} 6105 6106UPDATED 2002-07-10: As of @code{flex} version 2.5.9, this leak means that you did not 6107call @code{yylex_destroy()}. If you are using an earlier version of @code{flex}, then read 6108on. 6109 6110The leak is about 16426 bytes. That is, (8192 * 2 + 2) for the read-buffer, and 6111about 40 for @code{struct yy_buffer_state} (depending upon alignment). The leak is in 6112the non-reentrant C scanner only (NOT in the reentrant scanner, NOT in the C++ 6113scanner). Since @code{flex} doesn't know when you are done, the buffer is never freed. 6114 6115However, the leak won't multiply since the buffer is reused no matter how many 6116times you call @code{yylex()}. 6117 6118If you want to reclaim the memory when you are completely done scanning, then 6119you might try this: 6120 6121@example 6122@verbatim 6123/* For non-reentrant C scanner only. */ 6124yy_delete_buffer(YY_CURRENT_BUFFER); 6125yy_init = 1; 6126@end verbatim 6127@end example 6128 6129Note: @code{yy_init} is an "internal variable", and hasn't been tested in this 6130situation. It is possible that some other globals may need resetting as well. 6131 6132@node How do I track the byte offset for lseek()? 6133@unnumberedsec How do I track the byte offset for lseek()? 6134 6135@example 6136@verbatim 6137> We thought that it would be possible to have this number through the 6138> evaluation of the following expression: 6139> 6140> seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf 6141@end verbatim 6142@end example 6143 6144While this is the right idea, it has two problems. The first is that 6145it's possible that @code{flex} will request less than @code{YY_READ_BUF_SIZE} during 6146an invocation of @code{YY_INPUT} (or that your input source will return less 6147even though @code{YY_READ_BUF_SIZE} bytes were requested). The second problem 6148is that when refilling its internal buffer, @code{flex} keeps some characters 6149from the previous buffer (because usually it's in the middle of a match, 6150and needs those characters to construct @code{yytext} for the match once it's 6151done). Because of this, @code{yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf} won't 6152be exactly the number of characters already read from the current buffer. 6153 6154An alternative solution is to count the number of characters you've matched 6155since starting to scan. This can be done by using @code{YY_USER_ACTION}. For 6156example, 6157 6158@example 6159@verbatim 6160#define YY_USER_ACTION num_chars += yyleng; 6161@end verbatim 6162@end example 6163 6164(You need to be careful to update your bookkeeping if you use @code{yymore(}), 6165@code{yyless()}, @code{unput()}, or @code{input()}.) 6166 6167@node How do I use my own I/O classes in a C++ scanner? 6168@section How do I use my own I/O classes in a C++ scanner? 6169 6170When the flex C++ scanning class rewrite finally happens, then this sort of thing should become much easier. 6171 6172@cindex LexerOutput, overriding 6173@cindex LexerInput, overriding 6174@cindex overriding LexerOutput 6175@cindex overriding LexerInput 6176@cindex customizing I/O in C++ scanners 6177@cindex C++ I/O, customizing 6178You can do this by passing the various functions (such as @code{LexerInput()} 6179and @code{LexerOutput()}) NULL @code{iostream*}'s, and then 6180dealing with your own I/O classes surreptitiously (i.e., stashing them in 6181special member variables). This works because the only assumption about 6182the lexer regarding what's done with the iostream's is that they're 6183ultimately passed to @code{LexerInput()} and @code{LexerOutput}, which then do whatever 6184is necessary with them. 6185 6186@c faq edit stopped here 6187@node How do I skip as many chars as possible? 6188@unnumberedsec How do I skip as many chars as possible? 6189 6190How do I skip as many chars as possible -- without interfering with the other 6191patterns? 6192 6193In the example below, we want to skip over characters until we see the phrase 6194"endskip". The following will @emph{NOT} work correctly (do you see why not?) 6195 6196@example 6197@verbatim 6198/* INCORRECT SCANNER */ 6199%x SKIP 6200%% 6201<INITIAL>startskip BEGIN(SKIP); 6202... 6203<SKIP>"endskip" BEGIN(INITIAL); 6204<SKIP>.* ; 6205@end verbatim 6206@end example 6207 6208The problem is that the pattern .* will eat up the word "endskip." 6209The simplest (but slow) fix is: 6210 6211@example 6212@verbatim 6213<SKIP>"endskip" BEGIN(INITIAL); 6214<SKIP>. ; 6215@end verbatim 6216@end example 6217 6218The fix involves making the second rule match more, without 6219making it match "endskip" plus something else. So for example: 6220 6221@example 6222@verbatim 6223<SKIP>"endskip" BEGIN(INITIAL); 6224<SKIP>[^e]+ ; 6225<SKIP>. ;/* so you eat up e's, too */ 6226@end verbatim 6227@end example 6228 6229@c TODO: Evaluate this faq. 6230@node deleteme00 6231@unnumberedsec deleteme00 6232@example 6233@verbatim 6234QUESTION: 6235When was flex born? 6236 6237Vern Paxson took over 6238the Software Tools lex project from Jef Poskanzer in 1982. At that point it 6239was written in Ratfor. Around 1987 or so, Paxson translated it into C, and 6240a legend was born :-). 6241@end verbatim 6242@end example 6243 6244@c TODO: Evaluate this faq. 6245@node Are certain equivalent patterns faster than others? 6246@unnumberedsec Are certain equivalent patterns faster than others? 6247@example 6248@verbatim 6249To: Adoram Rogel <adoram@orna.hybridge.com> 6250Subject: Re: Flex 2.5.2 performance questions 6251In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT. 6252Date: Wed, 18 Sep 96 10:51:02 PDT 6253From: Vern Paxson <vern> 6254 6255[Note, the most recent flex release is 2.5.4, which you can get from 6256ftp.ee.lbl.gov. It has bug fixes over 2.5.2 and 2.5.3.] 6257 6258> 1. Using the pattern 6259> ([Ff](oot)?)?[Nn](ote)?(\.)? 6260> instead of 6261> (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.))) 6262> (in a very complicated flex program) caused the program to slow from 6263> 300K+/min to 100K/min (no other changes were done). 6264 6265These two are not equivalent. For example, the first can match "footnote." 6266but the second can only match "footnote". This is almost certainly the 6267cause in the discrepancy - the slower scanner run is matching more tokens, 6268and/or having to do more backing up. 6269 6270> 2. Which of these two are better: [Ff]oot or (F|f)oot ? 6271 6272From a performance point of view, they're equivalent (modulo presumably 6273minor effects such as memory cache hit rates; and the presence of trailing 6274context, see below). From a space point of view, the first is slightly 6275preferable. 6276 6277> 3. I have a pattern that look like this: 6278> pats {p1}|{p2}|{p3}|...|{p50} (50 patterns ORd) 6279> 6280> running yet another complicated program that includes the following rule: 6281> <snext>{and}/{no4}{bb}{pats} 6282> 6283> gets me to "too complicated - over 32,000 states"... 6284 6285I can't tell from this example whether the trailing context is variable-length 6286or fixed-length (it could be the latter if {and} is fixed-length). If it's 6287variable length, which flex -p will tell you, then this reflects a basic 6288performance problem, and if you can eliminate it by restructuring your 6289scanner, you will see significant improvement. 6290 6291> so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about 6292> 10 patterns and changed the rule to be 5 rules. 6293> This did compile, but what is the rule of thumb here ? 6294 6295The rule is to avoid trailing context other than fixed-length, in which for 6296a/b, either the 'a' pattern or the 'b' pattern have a fixed length. Use 6297of the '|' operator automatically makes the pattern variable length, so in 6298this case '[Ff]oot' is preferred to '(F|f)oot'. 6299 6300> 4. I changed a rule that looked like this: 6301> <snext8>{and}{bb}/{ROMAN}[^A-Za-z] { BEGIN... 6302> 6303> to the next 2 rules: 6304> <snext8>{and}{bb}/{ROMAN}[A-Za-z] { ECHO;} 6305> <snext8>{and}{bb}/{ROMAN} { BEGIN... 6306> 6307> Again, I understand the using [^...] will cause a great performance loss 6308 6309Actually, it doesn't cause any sort of performance loss. It's a surprising 6310fact about regular expressions that they always match in linear time 6311regardless of how complex they are. 6312 6313> but are there any specific rules about it ? 6314 6315See the "Performance Considerations" section of the man page, and also 6316the example in MISC/fastwc/. 6317 6318 Vern 6319@end verbatim 6320@end example 6321 6322@c TODO: Evaluate this faq. 6323@node Is backing up a big deal? 6324@unnumberedsec Is backing up a big deal? 6325@example 6326@verbatim 6327To: Adoram Rogel <adoram@hybridge.com> 6328Subject: Re: Flex 2.5.2 performance questions 6329In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT. 6330Date: Thu, 19 Sep 96 09:58:00 PDT 6331From: Vern Paxson <vern> 6332 6333> a lot about the backing up problem. 6334> I believe that there lies my biggest problem, and I'll try to improve 6335> it. 6336 6337Since you have variable trailing context, this is a bigger performance 6338problem. Fixing it is usually easier than fixing backing up, which in a 6339complicated scanner (yours seems to fit the bill) can be extremely 6340difficult to do correctly. 6341 6342You also don't mention what flags you are using for your scanner. 6343-f makes a large speed difference, and -Cfe buys you nearly as much 6344speed but the resulting scanner is considerably smaller. 6345 6346> I have an | operator in {and} and in {pats} so both of them are variable 6347> length. 6348 6349-p should have reported this. 6350 6351> Is changing one of them to fixed-length is enough ? 6352 6353Yes. 6354 6355> Is it possible to change the 32,000 states limit ? 6356 6357Yes. I've appended instructions on how. Before you make this change, 6358though, you should think about whether there are ways to fundamentally 6359simplify your scanner - those are certainly preferable! 6360 6361 Vern 6362 6363To increase the 32K limit (on a machine with 32 bit integers), you increase 6364the magnitude of the following in flexdef.h: 6365 6366#define JAMSTATE -32766 /* marks a reference to the state that always jams */ 6367#define MAXIMUM_MNS 31999 6368#define BAD_SUBSCRIPT -32767 6369#define MAX_SHORT 32700 6370 6371Adding a 0 or two after each should do the trick. 6372@end verbatim 6373@end example 6374 6375@c TODO: Evaluate this faq. 6376@node Can I fake multi-byte character support? 6377@unnumberedsec Can I fake multi-byte character support? 6378@example 6379@verbatim 6380To: Heeman_Lee@hp.com 6381Subject: Re: flex - multi-byte support? 6382In-reply-to: Your message of Thu, 03 Oct 1996 17:24:04 PDT. 6383Date: Fri, 04 Oct 1996 11:42:18 PDT 6384From: Vern Paxson <vern> 6385 6386> I assume as long as my *.l file defines the 6387> range of expected character code values (in octal format), flex will 6388> scan the file and read multi-byte characters correctly. But I have no 6389> confidence in this assumption. 6390 6391Your lack of confidence is justified - this won't work. 6392 6393Flex has in it a widespread assumption that the input is processed 6394one byte at a time. Fixing this is on the to-do list, but is involved, 6395so it won't happen any time soon. In the interim, the best I can suggest 6396(unless you want to try fixing it yourself) is to write your rules in 6397terms of pairs of bytes, using definitions in the first section: 6398 6399 X \xfe\xc2 6400 ... 6401 %% 6402 foo{X}bar found_foo_fe_c2_bar(); 6403 6404etc. Definitely a pain - sorry about that. 6405 6406By the way, the email address you used for me is ancient, indicating you 6407have a very old version of flex. You can get the most recent, 2.5.4, from 6408ftp.ee.lbl.gov. 6409 6410 Vern 6411@end verbatim 6412@end example 6413 6414@c TODO: Evaluate this faq. 6415@node deleteme01 6416@unnumberedsec deleteme01 6417@example 6418@verbatim 6419To: moleary@primus.com 6420Subject: Re: Flex / Unicode compatibility question 6421In-reply-to: Your message of Tue, 22 Oct 1996 10:15:42 PDT. 6422Date: Tue, 22 Oct 1996 11:06:13 PDT 6423From: Vern Paxson <vern> 6424 6425Unfortunately flex at the moment has a widespread assumption within it 6426that characters are processed 8 bits at a time. I don't see any easy 6427fix for this (other than writing your rules in terms of double characters - 6428a pain). I also don't know of a wider lex, though you might try surfing 6429the Plan 9 stuff because I know it's a Unicode system, and also the PCCT 6430toolkit (try searching say Alta Vista for "Purdue Compiler Construction 6431Toolkit"). 6432 6433Fixing flex to handle wider characters is on the long-term to-do list. 6434But since flex is a strictly spare-time project these days, this probably 6435won't happen for quite a while, unless someone else does it first. 6436 6437 Vern 6438@end verbatim 6439@end example 6440 6441@c TODO: Evaluate this faq. 6442@node Can you discuss some flex internals? 6443@unnumberedsec Can you discuss some flex internals? 6444@example 6445@verbatim 6446To: Johan Linde <jl@theophys.kth.se> 6447Subject: Re: translation of flex 6448In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST. 6449Date: Mon, 11 Nov 1996 10:33:50 PST 6450From: Vern Paxson <vern> 6451 6452> I'm working for the Swedish team translating GNU program, and I'm currently 6453> working with flex. I have a few questions about some of the messages which 6454> I hope you can answer. 6455 6456All of the things you're wondering about, by the way, concerning flex 6457internals - probably the only person who understands what they mean in 6458English is me! So I wouldn't worry too much about getting them right. 6459That said ... 6460 6461> #: main.c:545 6462> msgid " %d protos created\n" 6463> 6464> Does proto mean prototype? 6465 6466Yes - prototypes of state compression tables. 6467 6468> #: main.c:539 6469> msgid " %d/%d (peak %d) template nxt-chk entries created\n" 6470> 6471> Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?) 6472> However, 'template next-check entries' doesn't make much sense to me. To be 6473> able to find a good translation I need to know a little bit more about it. 6474 6475There is a scheme in the Aho/Sethi/Ullman compiler book for compressing 6476scanner tables. It involves creating two pairs of tables. The first has 6477"base" and "default" entries, the second has "next" and "check" entries. 6478The "base" entry is indexed by the current state and yields an index into 6479the next/check table. The "default" entry gives what to do if the state 6480transition isn't found in next/check. The "next" entry gives the next 6481state to enter, but only if the "check" entry verifies that this entry is 6482correct for the current state. Flex creates templates of series of 6483next/check entries and then encodes differences from these templates as a 6484way to compress the tables. 6485 6486> #: main.c:533 6487> msgid " %d/%d base-def entries created\n" 6488> 6489> The same problem here for 'base-def'. 6490 6491See above. 6492 6493 Vern 6494@end verbatim 6495@end example 6496 6497@c TODO: Evaluate this faq. 6498@node unput() messes up yy_at_bol 6499@unnumberedsec unput() messes up yy_at_bol 6500@example 6501@verbatim 6502To: Xinying Li <xli@npac.syr.edu> 6503Subject: Re: FLEX ? 6504In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST. 6505Date: Wed, 13 Nov 1996 19:51:54 PST 6506From: Vern Paxson <vern> 6507 6508> "unput()" them to input flow, question occurs. If I do this after I scan 6509> a carriage, the variable "YY_CURRENT_BUFFER->yy_at_bol" is changed. That 6510> means the carriage flag has gone. 6511 6512You can control this by calling yy_set_bol(). It's described in the manual. 6513 6514> And if in pre-reading it goes to the end of file, is anything done 6515> to control the end of curren buffer and end of file? 6516 6517No, there's no way to put back an end-of-file. 6518 6519> By the way I am using flex 2.5.2 and using the "-l". 6520 6521The latest release is 2.5.4, by the way. It fixes some bugs in 2.5.2 and 65222.5.3. You can get it from ftp.ee.lbl.gov. 6523 6524 Vern 6525@end verbatim 6526@end example 6527 6528@c TODO: Evaluate this faq. 6529@node The | operator is not doing what I want 6530@unnumberedsec The | operator is not doing what I want 6531@example 6532@verbatim 6533To: Alain.ISSARD@st.com 6534Subject: Re: Start condition with FLEX 6535In-reply-to: Your message of Mon, 18 Nov 1996 09:45:02 PST. 6536Date: Mon, 18 Nov 1996 10:41:34 PST 6537From: Vern Paxson <vern> 6538 6539> I am not able to use the start condition scope and to use the | (OR) with 6540> rules having start conditions. 6541 6542The problem is that if you use '|' as a regular expression operator, for 6543example "a|b" meaning "match either 'a' or 'b'", then it must *not* have 6544any blanks around it. If you instead want the special '|' *action* (which 6545from your scanner appears to be the case), which is a way of giving two 6546different rules the same action: 6547 6548 foo | 6549 bar matched_foo_or_bar(); 6550 6551then '|' *must* be separated from the first rule by whitespace and *must* 6552be followed by a new line. You *cannot* write it as: 6553 6554 foo | bar matched_foo_or_bar(); 6555 6556even though you might think you could because yacc supports this syntax. 6557The reason for this unfortunately incompatibility is historical, but it's 6558unlikely to be changed. 6559 6560Your problems with start condition scope are simply due to syntax errors 6561from your use of '|' later confusing flex. 6562 6563Let me know if you still have problems. 6564 6565 Vern 6566@end verbatim 6567@end example 6568 6569@c TODO: Evaluate this faq. 6570@node Why can't flex understand this variable trailing context pattern? 6571@unnumberedsec Why can't flex understand this variable trailing context pattern? 6572@example 6573@verbatim 6574To: Gregory Margo <gmargo@newton.vip.best.com> 6575Subject: Re: flex-2.5.3 bug report 6576In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST. 6577Date: Sat, 23 Nov 1996 17:07:32 PST 6578From: Vern Paxson <vern> 6579 6580> Enclosed is a lex file that "real" lex will process, but I cannot get 6581> flex to process it. Could you try it and maybe point me in the right direction? 6582 6583Your problem is that some of the definitions in the scanner use the '/' 6584trailing context operator, and have it enclosed in ()'s. Flex does not 6585allow this operator to be enclosed in ()'s because doing so allows undefined 6586regular expressions such as "(a/b)+". So the solution is to remove the 6587parentheses. Note that you must also be building the scanner with the -l 6588option for AT&T lex compatibility. Without this option, flex automatically 6589encloses the definitions in parentheses. 6590 6591 Vern 6592@end verbatim 6593@end example 6594 6595@c TODO: Evaluate this faq. 6596@node The ^ operator isn't working 6597@unnumberedsec The ^ operator isn't working 6598@example 6599@verbatim 6600To: Thomas Hadig <hadig@toots.physik.rwth-aachen.de> 6601Subject: Re: Flex Bug ? 6602In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST. 6603Date: Tue, 26 Nov 1996 11:15:05 PST 6604From: Vern Paxson <vern> 6605 6606> In my lexer code, i have the line : 6607> ^\*.* { } 6608> 6609> Thus all lines starting with an astrix (*) are comment lines. 6610> This does not work ! 6611 6612I can't get this problem to reproduce - it works fine for me. Note 6613though that if what you have is slightly different: 6614 6615 COMMENT ^\*.* 6616 %% 6617 {COMMENT} { } 6618 6619then it won't work, because flex pushes back macro definitions enclosed 6620in ()'s, so the rule becomes 6621 6622 (^\*.*) { } 6623 6624and now that the '^' operator is not at the immediate beginning of the 6625line, it's interpreted as just a regular character. You can avoid this 6626behavior by using the "-l" lex-compatibility flag, or "%option lex-compat". 6627 6628 Vern 6629@end verbatim 6630@end example 6631 6632@c TODO: Evaluate this faq. 6633@node Trailing context is getting confused with trailing optional patterns 6634@unnumberedsec Trailing context is getting confused with trailing optional patterns 6635@example 6636@verbatim 6637To: Adoram Rogel <adoram@hybridge.com> 6638Subject: Re: Flex 2.5.4 BOF ??? 6639In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST. 6640Date: Wed, 27 Nov 1996 10:56:25 PST 6641From: Vern Paxson <vern> 6642 6643> Organization(s)?/[a-z] 6644> 6645> This matched "Organizations" (looking in debug mode, the trailing s 6646> was matched with trailing context instead of the optional (s) in the 6647> end of the word. 6648 6649That should only happen with lex. Flex can properly match this pattern. 6650(That might be what you're saying, I'm just not sure.) 6651 6652> Is there a way to avoid this dangerous trailing context problem ? 6653 6654Unfortunately, there's no easy way. On the other hand, I don't see why 6655it should be a problem. Lex's matching is clearly wrong, and I'd hope 6656that usually the intent remains the same as expressed with the pattern, 6657so flex's matching will be correct. 6658 6659 Vern 6660@end verbatim 6661@end example 6662 6663@c TODO: Evaluate this faq. 6664@node Is flex GNU or not? 6665@unnumberedsec Is flex GNU or not? 6666@example 6667@verbatim 6668To: Cameron MacKinnon <mackin@interlog.com> 6669Subject: Re: Flex documentation bug 6670In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST. 6671Date: Sun, 01 Dec 1996 22:29:39 PST 6672From: Vern Paxson <vern> 6673 6674> I'm not sure how or where to submit bug reports (documentation or 6675> otherwise) for the GNU project stuff ... 6676 6677Well, strictly speaking flex isn't part of the GNU project. They just 6678distribute it because no one's written a decent GPL'd lex replacement. 6679So you should send bugs directly to me. Those sent to the GNU folks 6680sometimes find there way to me, but some may drop between the cracks. 6681 6682> In GNU Info, under the section 'Start Conditions', and also in the man 6683> page (mine's dated April '95) is a nice little snippet showing how to 6684> parse C quoted strings into a buffer, defined to be MAX_STR_CONST in 6685> size. Unfortunately, no overflow checking is ever done ... 6686 6687This is already mentioned in the manual: 6688 6689Finally, here's an example of how to match C-style quoted 6690strings using exclusive start conditions, including expanded 6691escape sequences (but not including checking for a string 6692that's too long): 6693 6694The reason for not doing the overflow checking is that it will needlessly 6695clutter up an example whose main purpose is just to demonstrate how to 6696use flex. 6697 6698The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov. 6699 6700 Vern 6701@end verbatim 6702@end example 6703 6704@c TODO: Evaluate this faq. 6705@node ERASEME53 6706@unnumberedsec ERASEME53 6707@example 6708@verbatim 6709To: tsv@cs.UManitoba.CA 6710Subject: Re: Flex (reg).. 6711In-reply-to: Your message of Thu, 06 Mar 1997 23:50:16 PST. 6712Date: Thu, 06 Mar 1997 15:54:19 PST 6713From: Vern Paxson <vern> 6714 6715> [:alpha:] ([:alnum:] | \\_)* 6716 6717If your rule really has embedded blanks as shown above, then it won't 6718work, as the first blank delimits the rule from the action. (It wouldn't 6719even compile ...) You need instead: 6720 6721[:alpha:]([:alnum:]|\\_)* 6722 6723and that should work fine - there's no restriction on what can go inside 6724of ()'s except for the trailing context operator, '/'. 6725 6726 Vern 6727@end verbatim 6728@end example 6729 6730@c TODO: Evaluate this faq. 6731@node I need to scan if-then-else blocks and while loops 6732@unnumberedsec I need to scan if-then-else blocks and while loops 6733@example 6734@verbatim 6735To: "Mike Stolnicki" <mstolnic@ford.com> 6736Subject: Re: FLEX help 6737In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT. 6738Date: Fri, 30 May 1997 10:46:35 PDT 6739From: Vern Paxson <vern> 6740 6741> We'd like to add "if-then-else", "while", and "for" statements to our 6742> language ... 6743> We've investigated many possible solutions. The one solution that seems 6744> the most reasonable involves knowing the position of a TOKEN in yyin. 6745 6746I strongly advise you to instead build a parse tree (abstract syntax tree) 6747and loop over that instead. You'll find this has major benefits in keeping 6748your interpreter simple and extensible. 6749 6750That said, the functionality you mention for get_position and set_position 6751have been on the to-do list for a while. As flex is a purely spare-time 6752project for me, no guarantees when this will be added (in particular, it 6753for sure won't be for many months to come). 6754 6755 Vern 6756@end verbatim 6757@end example 6758 6759@c TODO: Evaluate this faq. 6760@node ERASEME55 6761@unnumberedsec ERASEME55 6762@example 6763@verbatim 6764To: Colin Paul Adams <colin@colina.demon.co.uk> 6765Subject: Re: Flex C++ classes and Bison 6766In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT. 6767Date: Fri, 15 Aug 1997 10:48:19 PDT 6768From: Vern Paxson <vern> 6769 6770> #define YY_DECL int yylex (YYSTYPE *lvalp, struct parser_control 6771> *parm) 6772> 6773> I have been trying to get this to work as a C++ scanner, but it does 6774> not appear to be possible (warning that it matches no declarations in 6775> yyFlexLexer, or something like that). 6776> 6777> Is this supposed to be possible, or is it being worked on (I DID 6778> notice the comment that scanner classes are still experimental, so I'm 6779> not too hopeful)? 6780 6781What you need to do is derive a subclass from yyFlexLexer that provides 6782the above yylex() method, squirrels away lvalp and parm into member 6783variables, and then invokes yyFlexLexer::yylex() to do the regular scanning. 6784 6785 Vern 6786@end verbatim 6787@end example 6788 6789@c TODO: Evaluate this faq. 6790@node ERASEME56 6791@unnumberedsec ERASEME56 6792@example 6793@verbatim 6794To: Mikael.Latvala@lmf.ericsson.se 6795Subject: Re: Possible mistake in Flex v2.5 document 6796In-reply-to: Your message of Fri, 05 Sep 1997 16:07:24 PDT. 6797Date: Fri, 05 Sep 1997 10:01:54 PDT 6798From: Vern Paxson <vern> 6799 6800> In that example you show how to count comment lines when using 6801> C style /* ... */ comments. My question is, shouldn't you take into 6802> account a scenario where end of a comment marker occurs inside 6803> character or string literals? 6804 6805The scanner certainly needs to also scan character and string literals. 6806However it does that (there's an example in the man page for strings), the 6807lexer will recognize the beginning of the literal before it runs across the 6808embedded "/*". Consequently, it will finish scanning the literal before it 6809even considers the possibility of matching "/*". 6810 6811Example: 6812 6813 '([^']*|{ESCAPE_SEQUENCE})' 6814 6815will match all the text between the ''s (inclusive). So the lexer 6816considers this as a token beginning at the first ', and doesn't even 6817attempt to match other tokens inside it. 6818 6819I thinnk this subtlety is not worth putting in the manual, as I suspect 6820it would confuse more people than it would enlighten. 6821 6822 Vern 6823@end verbatim 6824@end example 6825 6826@c TODO: Evaluate this faq. 6827@node ERASEME57 6828@unnumberedsec ERASEME57 6829@example 6830@verbatim 6831To: "Marty Leisner" <leisner@sdsp.mc.xerox.com> 6832Subject: Re: flex limitations 6833In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT. 6834Date: Mon, 08 Sep 1997 11:38:08 PDT 6835From: Vern Paxson <vern> 6836 6837> %% 6838> [a-zA-Z]+ /* skip a line */ 6839> { printf("got %s\n", yytext); } 6840> %% 6841 6842What version of flex are you using? If I feed this to 2.5.4, it complains: 6843 6844 "bug.l", line 5: EOF encountered inside an action 6845 "bug.l", line 5: unrecognized rule 6846 "bug.l", line 5: fatal parse error 6847 6848Not the world's greatest error message, but it manages to flag the problem. 6849 6850(With the introduction of start condition scopes, flex can't accommodate 6851an action on a separate line, since it's ambiguous with an indented rule.) 6852 6853You can get 2.5.4 from ftp.ee.lbl.gov. 6854 6855 Vern 6856@end verbatim 6857@end example 6858 6859@c TODO: Evaluate this faq. 6860@node Is there a repository for flex scanners? 6861@unnumberedsec Is there a repository for flex scanners? 6862 6863Not that we know of. You might try asking on comp.compilers. 6864 6865@c TODO: Evaluate this faq. 6866@node How can I conditionally compile or preprocess my flex input file? 6867@unnumberedsec How can I conditionally compile or preprocess my flex input file? 6868 6869 6870Flex doesn't have a preprocessor like C does. You might try using m4, or the C 6871preprocessor plus a sed script to clean up the result. 6872 6873 6874@c TODO: Evaluate this faq. 6875@node Where can I find grammars for lex and yacc? 6876@unnumberedsec Where can I find grammars for lex and yacc? 6877 6878In the sources for flex and bison. 6879 6880@c TODO: Evaluate this faq. 6881@node I get an end-of-buffer message for each character scanned. 6882@unnumberedsec I get an end-of-buffer message for each character scanned. 6883 6884This will happen if your LexerInput() function returns only one character 6885at a time, which can happen either if you're scanner is "interactive", or 6886if the streams library on your platform always returns 1 for yyin->gcount(). 6887 6888Solution: override LexerInput() with a version that returns whole buffers. 6889 6890@c TODO: Evaluate this faq. 6891@node unnamed-faq-62 6892@unnumberedsec unnamed-faq-62 6893@example 6894@verbatim 6895To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 6896Subject: Re: Flex maximums 6897In-reply-to: Your message of Mon, 17 Nov 1997 17:16:06 PST. 6898Date: Mon, 17 Nov 1997 17:16:15 PST 6899From: Vern Paxson <vern> 6900 6901> I took a quick look into the flex-sources and altered some #defines in 6902> flexdefs.h: 6903> 6904> #define INITIAL_MNS 64000 6905> #define MNS_INCREMENT 1024000 6906> #define MAXIMUM_MNS 64000 6907 6908The things to fix are to add a couple of zeroes to: 6909 6910#define JAMSTATE -32766 /* marks a reference to the state that always jams */ 6911#define MAXIMUM_MNS 31999 6912#define BAD_SUBSCRIPT -32767 6913#define MAX_SHORT 32700 6914 6915and, if you get complaints about too many rules, make the following change too: 6916 6917 #define YY_TRAILING_MASK 0x200000 6918 #define YY_TRAILING_HEAD_MASK 0x400000 6919 6920- Vern 6921@end verbatim 6922@end example 6923 6924@c TODO: Evaluate this faq. 6925@node unnamed-faq-63 6926@unnumberedsec unnamed-faq-63 6927@example 6928@verbatim 6929To: jimmey@lexis-nexis.com (Jimmey Todd) 6930Subject: Re: FLEX question regarding istream vs ifstream 6931In-reply-to: Your message of Mon, 08 Dec 1997 15:54:15 PST. 6932Date: Mon, 15 Dec 1997 13:21:35 PST 6933From: Vern Paxson <vern> 6934 6935> stdin_handle = YY_CURRENT_BUFFER; 6936> ifstream fin( "aFile" ); 6937> yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) ); 6938> 6939> What I'm wanting to do, is pass the contents of a file thru one set 6940> of rules and then pass stdin thru another set... It works great if, I 6941> don't use the C++ classes. But since everything else that I'm doing is 6942> in C++, I thought I'd be consistent. 6943> 6944> The problem is that 'yy_create_buffer' is expecting an istream* as it's 6945> first argument (as stated in the man page). However, fin is a ifstream 6946> object. Any ideas on what I might be doing wrong? Any help would be 6947> appreciated. Thanks!! 6948 6949You need to pass &fin, to turn it into an ifstream* instead of an ifstream. 6950Then its type will be compatible with the expected istream*, because ifstream 6951is derived from istream. 6952 6953 Vern 6954@end verbatim 6955@end example 6956 6957@c TODO: Evaluate this faq. 6958@node unnamed-faq-64 6959@unnumberedsec unnamed-faq-64 6960@example 6961@verbatim 6962To: Enda Fadian <fadiane@piercom.ie> 6963Subject: Re: Question related to Flex man page? 6964In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST. 6965Date: Tue, 16 Dec 1997 14:17:09 PST 6966From: Vern Paxson <vern> 6967 6968> Can you explain to me what is ment by a long-jump in relation to flex? 6969 6970Using the longjmp() function while inside yylex() or a routine called by it. 6971 6972> what is the flex activation frame. 6973 6974Just yylex()'s stack frame. 6975 6976> As far as I can see yyrestart will bring me back to the sart of the input 6977> file and using flex++ isnot really an option! 6978 6979No, yyrestart() doesn't imply a rewind, even though its name might sound 6980like it does. It tells the scanner to flush its internal buffers and 6981start reading from the given file at its present location. 6982 6983 Vern 6984@end verbatim 6985@end example 6986 6987@c TODO: Evaluate this faq. 6988@node unnamed-faq-65 6989@unnumberedsec unnamed-faq-65 6990@example 6991@verbatim 6992To: hassan@larc.info.uqam.ca (Hassan Alaoui) 6993Subject: Re: Need urgent Help 6994In-reply-to: Your message of Sat, 20 Dec 1997 19:38:19 PST. 6995Date: Sun, 21 Dec 1997 21:30:46 PST 6996From: Vern Paxson <vern> 6997 6998> /usr/lib/yaccpar: In function `int yyparse()': 6999> /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)' 7000> 7001> ld: Undefined symbol 7002> _yylex 7003> _yyparse 7004> _yyin 7005 7006This is a known problem with Solaris C++ (and/or Solaris yacc). I believe 7007the fix is to explicitly insert some 'extern "C"' statements for the 7008corresponding routines/symbols. 7009 7010 Vern 7011@end verbatim 7012@end example 7013 7014@c TODO: Evaluate this faq. 7015@node unnamed-faq-66 7016@unnumberedsec unnamed-faq-66 7017@example 7018@verbatim 7019To: mc0307@mclink.it 7020Cc: gnu@prep.ai.mit.edu 7021Subject: Re: [mc0307@mclink.it: Help request] 7022In-reply-to: Your message of Fri, 12 Dec 1997 17:57:29 PST. 7023Date: Sun, 21 Dec 1997 22:33:37 PST 7024From: Vern Paxson <vern> 7025 7026> This is my definition for float and integer types: 7027> . . . 7028> NZD [1-9] 7029> ... 7030> I've tested my program on other lex version (on UNIX Sun Solaris an HP 7031> UNIX) and it work well, so I think that my definitions are correct. 7032> There are any differences between Lex and Flex? 7033 7034There are indeed differences, as discussed in the man page. The one 7035you are probably running into is that when flex expands a name definition, 7036it puts parentheses around the expansion, while lex does not. There's 7037an example in the man page of how this can lead to different matching. 7038Flex's behavior complies with the POSIX standard (or at least with the 7039last POSIX draft I saw). 7040 7041 Vern 7042@end verbatim 7043@end example 7044 7045@c TODO: Evaluate this faq. 7046@node unnamed-faq-67 7047@unnumberedsec unnamed-faq-67 7048@example 7049@verbatim 7050To: hassan@larc.info.uqam.ca (Hassan Alaoui) 7051Subject: Re: Thanks 7052In-reply-to: Your message of Mon, 22 Dec 1997 16:06:35 PST. 7053Date: Mon, 22 Dec 1997 14:35:05 PST 7054From: Vern Paxson <vern> 7055 7056> Thank you very much for your help. I compile and link well with C++ while 7057> declaring 'yylex ...' extern, But a little problem remains. I get a 7058> segmentation default when executing ( I linked with lfl library) while it 7059> works well when using LEX instead of flex. Do you have some ideas about the 7060> reason for this ? 7061 7062The one possible reason for this that comes to mind is if you've defined 7063yytext as "extern char yytext[]" (which is what lex uses) instead of 7064"extern char *yytext" (which is what flex uses). If it's not that, then 7065I'm afraid I don't know what the problem might be. 7066 7067 Vern 7068@end verbatim 7069@end example 7070 7071@c TODO: Evaluate this faq. 7072@node unnamed-faq-68 7073@unnumberedsec unnamed-faq-68 7074@example 7075@verbatim 7076To: "Bart Niswonger" <NISWONGR@almaden.ibm.com> 7077Subject: Re: flex 2.5: c++ scanners & start conditions 7078In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST. 7079Date: Tue, 06 Jan 1998 19:19:30 PST 7080From: Vern Paxson <vern> 7081 7082> The problem is that when I do this (using %option c++) start 7083> conditions seem to not apply. 7084 7085The BEGIN macro modifies the yy_start variable. For C scanners, this 7086is a static with scope visible through the whole file. For C++ scanners, 7087it's a member variable, so it only has visible scope within a member 7088function. Your lexbegin() routine is not a member function when you 7089build a C++ scanner, so it's not modifying the correct yy_start. The 7090diagnostic that indicates this is that you found you needed to add 7091a declaration of yy_start in order to get your scanner to compile when 7092using C++; instead, the correct fix is to make lexbegin() a member 7093function (by deriving from yyFlexLexer). 7094 7095 Vern 7096@end verbatim 7097@end example 7098 7099@c TODO: Evaluate this faq. 7100@node unnamed-faq-69 7101@unnumberedsec unnamed-faq-69 7102@example 7103@verbatim 7104To: "Boris Zinin" <boris@ippe.rssi.ru> 7105Subject: Re: current position in flex buffer 7106In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST. 7107Date: Mon, 12 Jan 1998 12:03:15 PST 7108From: Vern Paxson <vern> 7109 7110> The problem is how to determine the current position in flex active 7111> buffer when a rule is matched.... 7112 7113You will need to keep track of this explicitly, such as by redefining 7114YY_USER_ACTION to count the number of characters matched. 7115 7116The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov. 7117 7118 Vern 7119@end verbatim 7120@end example 7121 7122@c TODO: Evaluate this faq. 7123@node unnamed-faq-70 7124@unnumberedsec unnamed-faq-70 7125@example 7126@verbatim 7127To: Bik.Dhaliwal@bis.org 7128Subject: Re: Flex question 7129In-reply-to: Your message of Mon, 26 Jan 1998 13:05:35 PST. 7130Date: Tue, 27 Jan 1998 22:41:52 PST 7131From: Vern Paxson <vern> 7132 7133> That requirement involves knowing 7134> the character position at which a particular token was matched 7135> in the lexer. 7136 7137The way you have to do this is by explicitly keeping track of where 7138you are in the file, by counting the number of characters scanned 7139for each token (available in yyleng). It may prove convenient to 7140do this by redefining YY_USER_ACTION, as described in the manual. 7141 7142 Vern 7143@end verbatim 7144@end example 7145 7146@c TODO: Evaluate this faq. 7147@node unnamed-faq-71 7148@unnumberedsec unnamed-faq-71 7149@example 7150@verbatim 7151To: Vladimir Alexiev <vladimir@cs.ualberta.ca> 7152Subject: Re: flex: how to control start condition from parser? 7153In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST. 7154Date: Tue, 27 Jan 1998 22:45:37 PST 7155From: Vern Paxson <vern> 7156 7157> It seems useful for the parser to be able to tell the lexer about such 7158> context dependencies, because then they don't have to be limited to 7159> local or sequential context. 7160 7161One way to do this is to have the parser call a stub routine that's 7162included in the scanner's .l file, and consequently that has access ot 7163BEGIN. The only ugliness is that the parser can't pass in the state 7164it wants, because those aren't visible - but if you don't have many 7165such states, then using a different set of names doesn't seem like 7166to much of a burden. 7167 7168While generating a .h file like you suggests is certainly cleaner, 7169flex development has come to a virtual stand-still :-(, so a workaround 7170like the above is much more pragmatic than waiting for a new feature. 7171 7172 Vern 7173@end verbatim 7174@end example 7175 7176@c TODO: Evaluate this faq. 7177@node unnamed-faq-72 7178@unnumberedsec unnamed-faq-72 7179@example 7180@verbatim 7181To: Barbara Denny <denny@3com.com> 7182Subject: Re: freebsd flex bug? 7183In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST. 7184Date: Fri, 30 Jan 1998 12:42:32 PST 7185From: Vern Paxson <vern> 7186 7187> lex.yy.c:1996: parse error before `=' 7188 7189This is the key, identifying this error. (It may help to pinpoint 7190it by using flex -L, so it doesn't generate #line directives in its 7191output.) I will bet you heavy money that you have a start condition 7192name that is also a variable name, or something like that; flex spits 7193out #define's for each start condition name, mapping them to a number, 7194so you can wind up with: 7195 7196 %x foo 7197 %% 7198 ... 7199 %% 7200 void bar() 7201 { 7202 int foo = 3; 7203 } 7204 7205and the penultimate will turn into "int 1 = 3" after C preprocessing, 7206since flex will put "#define foo 1" in the generated scanner. 7207 7208 Vern 7209@end verbatim 7210@end example 7211 7212@c TODO: Evaluate this faq. 7213@node unnamed-faq-73 7214@unnumberedsec unnamed-faq-73 7215@example 7216@verbatim 7217To: Maurice Petrie <mpetrie@infoscigroup.com> 7218Subject: Re: Lost flex .l file 7219In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST. 7220Date: Mon, 02 Feb 1998 11:15:12 PST 7221From: Vern Paxson <vern> 7222 7223> I am curious as to 7224> whether there is a simple way to backtrack from the generated source to 7225> reproduce the lost list of tokens we are searching on. 7226 7227In theory, it's straight-forward to go from the DFA representation 7228back to a regular-expression representation - the two are isomorphic. 7229In practice, a huge headache, because you have to unpack all the tables 7230back into a single DFA representation, and then write a program to munch 7231on that and translate it into an RE. 7232 7233Sorry for the less-than-happy news ... 7234 7235 Vern 7236@end verbatim 7237@end example 7238 7239@c TODO: Evaluate this faq. 7240@node unnamed-faq-74 7241@unnumberedsec unnamed-faq-74 7242@example 7243@verbatim 7244To: jimmey@lexis-nexis.com (Jimmey Todd) 7245Subject: Re: Flex performance question 7246In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. 7247Date: Thu, 19 Feb 1998 08:48:51 PST 7248From: Vern Paxson <vern> 7249 7250> What I have found, is that the smaller the data chunk, the faster the 7251> program executes. This is the opposite of what I expected. Should this be 7252> happening this way? 7253 7254This is exactly what will happen if your input file has embedded NULs. 7255From the man page: 7256 7257A final note: flex is slow when matching NUL's, particularly 7258when a token contains multiple NUL's. It's best to write 7259rules which match short amounts of text if it's anticipated 7260that the text will often include NUL's. 7261 7262So that's the first thing to look for. 7263 7264 Vern 7265@end verbatim 7266@end example 7267 7268@c TODO: Evaluate this faq. 7269@node unnamed-faq-75 7270@unnumberedsec unnamed-faq-75 7271@example 7272@verbatim 7273To: jimmey@lexis-nexis.com (Jimmey Todd) 7274Subject: Re: Flex performance question 7275In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. 7276Date: Thu, 19 Feb 1998 15:42:25 PST 7277From: Vern Paxson <vern> 7278 7279So there are several problems. 7280 7281First, to go fast, you want to match as much text as possible, which 7282your scanners don't in the case that what they're scanning is *not* 7283a <RN> tag. So you want a rule like: 7284 7285 [^<]+ 7286 7287Second, C++ scanners are particularly slow if they're interactive, 7288which they are by default. Using -B speeds it up by a factor of 3-4 7289on my workstation. 7290 7291Third, C++ scanners that use the istream interface are slow, because 7292of how poorly implemented istream's are. I built two versions of 7293the following scanner: 7294 7295 %% 7296 .*\n 7297 .* 7298 %% 7299 7300and the C version inhales a 2.5MB file on my workstation in 0.8 seconds. 7301The C++ istream version, using -B, takes 3.8 seconds. 7302 7303 Vern 7304@end verbatim 7305@end example 7306 7307@c TODO: Evaluate this faq. 7308@node unnamed-faq-76 7309@unnumberedsec unnamed-faq-76 7310@example 7311@verbatim 7312To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com> 7313Subject: Re: FLEX 2.5 & THE YEAR 2000 7314In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT. 7315Date: Wed, 03 Jun 1998 10:22:26 PDT 7316From: Vern Paxson <vern> 7317 7318> I am researching the Y2K problem with General Electric R&D 7319> and need to know if there are any known issues concerning 7320> the above mentioned software and Y2K regardless of version. 7321 7322There shouldn't be, all it ever does with the date is ask the system 7323for it and then print it out. 7324 7325 Vern 7326@end verbatim 7327@end example 7328 7329@c TODO: Evaluate this faq. 7330@node unnamed-faq-77 7331@unnumberedsec unnamed-faq-77 7332@example 7333@verbatim 7334To: "Hans Dermot Doran" <htd@ibhdoran.com> 7335Subject: Re: flex problem 7336In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT. 7337Date: Tue, 21 Jul 1998 14:23:34 PDT 7338From: Vern Paxson <vern> 7339 7340> To overcome this, I gets() the stdin into a string and lex the string. The 7341> string is lexed OK except that the end of string isn't lexed properly 7342> (yy_scan_string()), that is the lexer dosn't recognise the end of string. 7343 7344Flex doesn't contain mechanisms for recognizing buffer endpoints. But if 7345you use fgets instead (which you should anyway, to protect against buffer 7346overflows), then the final \n will be preserved in the string, and you can 7347scan that in order to find the end of the string. 7348 7349 Vern 7350@end verbatim 7351@end example 7352 7353@c TODO: Evaluate this faq. 7354@node unnamed-faq-78 7355@unnumberedsec unnamed-faq-78 7356@example 7357@verbatim 7358To: soumen@almaden.ibm.com 7359Subject: Re: Flex++ 2.5.3 instance member vs. static member 7360In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT. 7361Date: Tue, 28 Jul 1998 01:10:34 PDT 7362From: Vern Paxson <vern> 7363 7364> %{ 7365> int mylineno = 0; 7366> %} 7367> ws [ \t]+ 7368> alpha [A-Za-z] 7369> dig [0-9] 7370> %% 7371> 7372> Now you'd expect mylineno to be a member of each instance of class 7373> yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to 7374> indicate otherwise; unless I am missing something the declaration of 7375> mylineno seems to be outside any class scope. 7376> 7377> How will this work if I want to run a multi-threaded application with each 7378> thread creating a FlexLexer instance? 7379 7380Derive your own subclass and make mylineno a member variable of it. 7381 7382 Vern 7383@end verbatim 7384@end example 7385 7386@c TODO: Evaluate this faq. 7387@node unnamed-faq-79 7388@unnumberedsec unnamed-faq-79 7389@example 7390@verbatim 7391To: Adoram Rogel <adoram@hybridge.com> 7392Subject: Re: More than 32K states change hangs 7393In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT. 7394Date: Tue, 04 Aug 1998 22:28:45 PDT 7395From: Vern Paxson <vern> 7396 7397> Vern Paxson, 7398> 7399> I followed your advice, posted on Usenet bu you, and emailed to me 7400> personally by you, on how to overcome the 32K states limit. I'm running 7401> on Linux machines. 7402> I took the full source of version 2.5.4 and did the following changes in 7403> flexdef.h: 7404> #define JAMSTATE -327660 7405> #define MAXIMUM_MNS 319990 7406> #define BAD_SUBSCRIPT -327670 7407> #define MAX_SHORT 327000 7408> 7409> and compiled. 7410> All looked fine, including check and bigcheck, so I installed. 7411 7412Hmmm, you shouldn't increase MAX_SHORT, though looking through my email 7413archives I see that I did indeed recommend doing so. Try setting it back 7414to 32700; that should suffice that you no longer need -Ca. If it still 7415hangs, then the interesting question is - where? 7416 7417> Compiling the same hanged program with a out-of-the-box (RedHat 4.2 7418> distribution of Linux) 7419> flex 2.5.4 binary works. 7420 7421Since Linux comes with source code, you should diff it against what 7422you have to see what problems they missed. 7423 7424> Should I always compile with the -Ca option now ? even short and simple 7425> filters ? 7426 7427No, definitely not. It's meant to be for those situations where you 7428absolutely must squeeze every last cycle out of your scanner. 7429 7430 Vern 7431@end verbatim 7432@end example 7433 7434@c TODO: Evaluate this faq. 7435@node unnamed-faq-80 7436@unnumberedsec unnamed-faq-80 7437@example 7438@verbatim 7439To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com> 7440Subject: Re: flex output for static code portion 7441In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT. 7442Date: Mon, 17 Aug 1998 23:57:42 PDT 7443From: Vern Paxson <vern> 7444 7445> I would like to use flex under the hood to generate a binary file 7446> containing the data structures that control the parse. 7447 7448This has been on the wish-list for a long time. In principle it's 7449straight-forward - you redirect mkdata() et al's I/O to another file, 7450and modify the skeleton to have a start-up function that slurps these 7451into dynamic arrays. The concerns are (1) the scanner generation code 7452is hairy and full of corner cases, so it's easy to get surprised when 7453going down this path :-( ; and (2) being careful about buffering so 7454that when the tables change you make sure the scanner starts in the 7455correct state and reading at the right point in the input file. 7456 7457> I was wondering if you know of anyone who has used flex in this way. 7458 7459I don't - but it seems like a reasonable project to undertake (unlike 7460numerous other flex tweaks :-). 7461 7462 Vern 7463@end verbatim 7464@end example 7465 7466@c TODO: Evaluate this faq. 7467@node unnamed-faq-81 7468@unnumberedsec unnamed-faq-81 7469@example 7470@verbatim 7471Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11]) 7472 by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838 7473 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT) 7474Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2]) 7475 by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694 7476 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200 7477Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200 7478From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de> 7479Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de> 7480Subject: "flex scanner push-back overflow" 7481To: vern@ee.lbl.gov 7482Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST) 7483Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 7484X-NoJunk: Do NOT send commercial mail, spam or ads to this address! 7485X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/ 7486X-Mailer: ELM [version 2.4ME+ PL28 (25)] 7487MIME-Version: 1.0 7488Content-Type: text/plain; charset=US-ASCII 7489Content-Transfer-Encoding: 7bit 7490 7491Hi Vern, 7492 7493Yesterday, I encountered a strange problem: I use the macro processor m4 7494to include some lengthy lists into a .l file. Following is a flex macro 7495definition that causes some serious pain in my neck: 7496 7497AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...]) 7498 7499The complete list contains about 10kB. When I try to "flex" this file 7500(on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased 7501some of the predefined values in flexdefs.h) I get the error: 7502 7503myflex/flex -8 sentag.tmp.l 7504flex scanner push-back overflow 7505 7506When I remove the slashes in the macro definition everything works fine. 7507As I understand it, the double quotes escape the slash-character so it 7508really means "/" and not "trailing context". Furthermore, I tried to 7509escape the slashes with backslashes, but with no use, the same error message 7510appeared when flexing the code. 7511 7512Do you have an idea what's going on here? 7513 7514Greetings from Germany, 7515 Georg 7516-- 7517Georg Rehm georg@cl-ki.uni-osnabrueck.de 7518Institute for Semantic Information Processing, University of Osnabrueck, FRG 7519@end verbatim 7520@end example 7521 7522@c TODO: Evaluate this faq. 7523@node unnamed-faq-82 7524@unnumberedsec unnamed-faq-82 7525@example 7526@verbatim 7527To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 7528Subject: Re: "flex scanner push-back overflow" 7529In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT. 7530Date: Thu, 20 Aug 1998 07:05:35 PDT 7531From: Vern Paxson <vern> 7532 7533> myflex/flex -8 sentag.tmp.l 7534> flex scanner push-back overflow 7535 7536Flex itself uses a flex scanner. That scanner is running out of buffer 7537space when it tries to unput() the humongous macro you've defined. When 7538you remove the '/'s, you make it small enough so that it fits in the buffer; 7539removing spaces would do the same thing. 7540 7541The fix is to either rethink how come you're using such a big macro and 7542perhaps there's another/better way to do it; or to rebuild flex's own 7543scan.c with a larger value for 7544 7545 #define YY_BUF_SIZE 16384 7546 7547- Vern 7548@end verbatim 7549@end example 7550 7551@c TODO: Evaluate this faq. 7552@node unnamed-faq-83 7553@unnumberedsec unnamed-faq-83 7554@example 7555@verbatim 7556To: Jan Kort <jan@research.techforce.nl> 7557Subject: Re: Flex 7558In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200. 7559Date: Sat, 05 Sep 1998 00:59:49 PDT 7560From: Vern Paxson <vern> 7561 7562> %% 7563> 7564> "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); } 7565> ^\n { fprintf(stderr, "empty line\n"); } 7566> . { } 7567> \n { fprintf(stderr, "new line\n"); } 7568> 7569> %% 7570> -- input --------------------------------------- 7571> TEST1 7572> -- output -------------------------------------- 7573> TEST1 7574> empty line 7575> ------------------------------------------------ 7576 7577IMHO, it's not clear whether or not this is in fact a bug. It depends 7578on whether you view yyless() as backing up in the input stream, or as 7579pushing new characters onto the beginning of the input stream. Flex 7580interprets it as the latter (for implementation convenience, I'll admit), 7581and so considers the newline as in fact matching at the beginning of a 7582line, as after all the last token scanned an entire line and so the 7583scanner is now at the beginning of a new line. 7584 7585I agree that this is counter-intuitive for yyless(), given its 7586functional description (it's less so for unput(), depending on whether 7587you're unput()'ing new text or scanned text). But I don't plan to 7588change it any time soon, as it's a pain to do so. Consequently, 7589you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak 7590your scanner into the behavior you desire. 7591 7592Sorry for the less-than-completely-satisfactory answer. 7593 7594 Vern 7595@end verbatim 7596@end example 7597 7598@c TODO: Evaluate this faq. 7599@node unnamed-faq-84 7600@unnumberedsec unnamed-faq-84 7601@example 7602@verbatim 7603To: Patrick Krusenotto <krusenot@mac-info-link.de> 7604Subject: Re: Problems with restarting flex-2.5.2-generated scanner 7605In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT. 7606Date: Thu, 24 Sep 1998 23:28:43 PDT 7607From: Vern Paxson <vern> 7608 7609> I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately 7610> trying to make my scanner restart with a new file after my parser stops 7611> with a parse error. When my compiler restarts, the parser always 7612> receives the token after the token (in the old file!) that caused the 7613> parser error. 7614 7615I suspect the problem is that your parser has read ahead in order 7616to attempt to resolve an ambiguity, and when it's restarted it picks 7617up with that token rather than reading a fresh one. If you're using 7618yacc, then the special "error" production can sometimes be used to 7619consume tokens in an attempt to get the parser into a consistent state. 7620 7621 Vern 7622@end verbatim 7623@end example 7624 7625@c TODO: Evaluate this faq. 7626@node unnamed-faq-85 7627@unnumberedsec unnamed-faq-85 7628@example 7629@verbatim 7630To: Henric Jungheim <junghelh@pe-nelson.com> 7631Subject: Re: flex 2.5.4a 7632In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST. 7633Date: Tue, 27 Oct 1998 16:50:14 PST 7634From: Vern Paxson <vern> 7635 7636> This brings up a feature request: How about a command line 7637> option to specify the filename when reading from stdin? That way one 7638> doesn't need to create a temporary file in order to get the "#line" 7639> directives to make sense. 7640 7641Use -o combined with -t (per the man page description of -o). 7642 7643> P.S., Is there any simple way to use non-blocking IO to parse multiple 7644> streams? 7645 7646Simple, no. 7647 7648One approach might be to return a magic character on EWOULDBLOCK and 7649have a rule 7650 7651 .*<magic-character> // put back .*, eat magic character 7652 7653This is off the top of my head, not sure it'll work. 7654 7655 Vern 7656@end verbatim 7657@end example 7658 7659@c TODO: Evaluate this faq. 7660@node unnamed-faq-86 7661@unnumberedsec unnamed-faq-86 7662@example 7663@verbatim 7664To: "Repko, Billy D" <billy.d.repko@intel.com> 7665Subject: Re: Compiling scanners 7666In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST. 7667Date: Thu, 14 Jan 1999 00:25:30 PST 7668From: Vern Paxson <vern> 7669 7670> It appears that maybe it cannot find the lfl library. 7671 7672The Makefile in the distribution builds it, so you should have it. 7673It's exceedingly trivial, just a main() that calls yylex() and 7674a yyrap() that always returns 1. 7675 7676> %% 7677> \n ++num_lines; ++num_chars; 7678> . ++num_chars; 7679 7680You can't indent your rules like this - that's where the errors are coming 7681from. Flex copies indented text to the output file, it's how you do things 7682like 7683 7684 int num_lines_seen = 0; 7685 7686to declare local variables. 7687 7688 Vern 7689@end verbatim 7690@end example 7691 7692@c TODO: Evaluate this faq. 7693@node unnamed-faq-87 7694@unnumberedsec unnamed-faq-87 7695@example 7696@verbatim 7697To: Erick Branderhorst <Erick.Branderhorst@asml.nl> 7698Subject: Re: flex input buffer 7699In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST. 7700Date: Tue, 09 Feb 1999 21:03:37 PST 7701From: Vern Paxson <vern> 7702 7703> In the flex.skl file the size of the default input buffers is set. Can you 7704> explain why this size is set and why it is such a high number. 7705 7706It's large to optimize performance when scanning large files. You can 7707safely make it a lot lower if needed. 7708 7709 Vern 7710@end verbatim 7711@end example 7712 7713@c TODO: Evaluate this faq. 7714@node unnamed-faq-88 7715@unnumberedsec unnamed-faq-88 7716@example 7717@verbatim 7718To: "Guido Minnen" <guidomi@cogs.susx.ac.uk> 7719Subject: Re: Flex error message 7720In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST. 7721Date: Thu, 25 Feb 1999 00:11:31 PST 7722From: Vern Paxson <vern> 7723 7724> I'm extending a larger scanner written in Flex and I keep running into 7725> problems. More specifically, I get the error message: 7726> "flex: input rules are too complicated (>= 32000 NFA states)" 7727 7728Increase the definitions in flexdef.h for: 7729 7730#define JAMSTATE -32766 /* marks a reference to the state that always j 7731ams */ 7732#define MAXIMUM_MNS 31999 7733#define BAD_SUBSCRIPT -32767 7734 7735recompile everything, and it should all work. 7736 7737 Vern 7738@end verbatim 7739@end example 7740 7741@c TODO: Evaluate this faq. 7742@node unnamed-faq-90 7743@unnumberedsec unnamed-faq-90 7744@example 7745@verbatim 7746To: "Dmitriy Goldobin" <gold@ems.chel.su> 7747Subject: Re: FLEX trouble 7748In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT. 7749Date: Tue, 01 Jun 1999 00:15:07 PDT 7750From: Vern Paxson <vern> 7751 7752> I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20 7753> but rule "/*"(.|\n)*"*/" don't work ? 7754 7755The second of these will have to scan the entire input stream (because 7756"(.|\n)*" matches an arbitrary amount of any text) in order to see if 7757it ends with "*/", terminating the comment. That potentially will overflow 7758the input buffer. 7759 7760> More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error 7761> 'unrecognized rule'. 7762 7763You can't use the '/' operator inside parentheses. It's not clear 7764what "(a/b)*" actually means. 7765 7766> I now use workaround with state <comment>, but single-rule is 7767> better, i think. 7768 7769Single-rule is nice but will always have the problem of either setting 7770restrictions on comments (like not allowing multi-line comments) and/or 7771running the risk of consuming the entire input stream, as noted above. 7772 7773 Vern 7774@end verbatim 7775@end example 7776 7777@c TODO: Evaluate this faq. 7778@node unnamed-faq-91 7779@unnumberedsec unnamed-faq-91 7780@example 7781@verbatim 7782Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18]) 7783 by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100 7784 for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT) 7785Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999 7786To: vern@ee.lbl.gov 7787Date: Tue, 15 Jun 1999 08:55:43 -0700 7788From: "Aki Niimura" <neko@my-deja.com> 7789Message-ID: <KNONDOHDOBGAEAAA@my-deja.com> 7790Mime-Version: 1.0 7791Cc: 7792X-Sent-Mail: on 7793Reply-To: 7794X-Mailer: MailCity Service 7795Subject: A question on flex C++ scanner 7796X-Sender-Ip: 12.72.207.61 7797Organization: My Deja Email (http://www.my-deja.com:80) 7798Content-Type: text/plain; charset=us-ascii 7799Content-Transfer-Encoding: 7bit 7800 7801Dear Dr. Paxon, 7802 7803I have been using flex for years. 7804It works very well on many projects. 7805Most case, I used it to generate a scanner on C language. 7806However, one project I needed to generate a scanner 7807on C++ lanuage. Thanks to your enhancement, flex did 7808the job. 7809 7810Currently, I'm working on enhancing my previous project. 7811I need to deal with multiple input streams (recursive 7812inclusion) in this scanner (C++). 7813I did similar thing for another scanner (C) as you 7814explained in your documentation. 7815 7816The generated scanner (C++) has necessary methods: 7817- switch_to_buffer(struct yy_buffer_state *b) 7818- yy_create_buffer(istream *is, int sz) 7819- yy_delete_buffer(struct yy_buffer_state *b) 7820 7821However, I couldn't figure out how to access current 7822buffer (yy_current_buffer). 7823 7824yy_current_buffer is a protected member of yyFlexLexer. 7825I can't access it directly. 7826Then, I thought yy_create_buffer() with is = 0 might 7827return current stream buffer. But it seems not as far 7828as I checked the source. (flex 2.5.4) 7829 7830I went through the Web in addition to Flex documentation. 7831However, it hasn't been successful, so far. 7832 7833It is not my intention to bother you, but, can you 7834comment about how to obtain the current stream buffer? 7835 7836Your response would be highly appreciated. 7837 7838Best regards, 7839Aki Niimura 7840 7841--== Sent via Deja.com http://www.deja.com/ ==-- 7842Share what you know. Learn what you don't. 7843@end verbatim 7844@end example 7845 7846@c TODO: Evaluate this faq. 7847@node unnamed-faq-92 7848@unnumberedsec unnamed-faq-92 7849@example 7850@verbatim 7851To: neko@my-deja.com 7852Subject: Re: A question on flex C++ scanner 7853In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT. 7854Date: Tue, 15 Jun 1999 09:04:24 PDT 7855From: Vern Paxson <vern> 7856 7857> However, I couldn't figure out how to access current 7858> buffer (yy_current_buffer). 7859 7860Derive your own subclass from yyFlexLexer. 7861 7862 Vern 7863@end verbatim 7864@end example 7865 7866@c TODO: Evaluate this faq. 7867@node unnamed-faq-93 7868@unnumberedsec unnamed-faq-93 7869@example 7870@verbatim 7871To: "Stones, Darren" <Darren.Stones@nectech.co.uk> 7872Subject: Re: You're the man to see? 7873In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT. 7874Date: Wed, 23 Jun 1999 09:01:40 PDT 7875From: Vern Paxson <vern> 7876 7877> I hope you can help me. I am using Flex and Bison to produce an interpreted 7878> language. However all goes well until I try to implement an IF statement or 7879> a WHILE. I cannot get this to work as the parser parses all the conditions 7880> eg. the TRUE and FALSE conditons to check for a rule match. So I cannot 7881> make a decision!! 7882 7883You need to use the parser to build a parse tree (= abstract syntax trwee), 7884and when that's all done you recursively evaluate the tree, binding variables 7885to values at that time. 7886 7887 Vern 7888@end verbatim 7889@end example 7890 7891@c TODO: Evaluate this faq. 7892@node unnamed-faq-94 7893@unnumberedsec unnamed-faq-94 7894@example 7895@verbatim 7896To: Petr Danecek <petr@ics.cas.cz> 7897Subject: Re: flex - question 7898In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT. 7899Date: Fri, 02 Jul 1999 16:52:13 PDT 7900From: Vern Paxson <vern> 7901 7902> file, it takes an enormous amount of time. It is funny, because the 7903> source code has only 12 rules!!! I think it looks like an exponencial 7904> growth. 7905 7906Right, that's the problem - some patterns (those with a lot of 7907ambiguity, where yours has because at any given time the scanner can 7908be in the middle of all sorts of combinations of the different 7909rules) blow up exponentially. 7910 7911For your rules, there is an easy fix. Change the ".*" that comes fater 7912the directory name to "[^ ]*". With that in place, the rules are no 7913longer nearly so ambiguous, because then once one of the directories 7914has been matched, no other can be matched (since they all require a 7915leading blank). 7916 7917If that's not an acceptable solution, then you can enter a start state 7918to pick up the .*\n after each directory is matched. 7919 7920Also note that for speed, you'll want to add a ".*" rule at the end, 7921otherwise rules that don't match any of the patterns will be matched 7922very slowly, a character at a time. 7923 7924 Vern 7925@end verbatim 7926@end example 7927 7928@c TODO: Evaluate this faq. 7929@node unnamed-faq-95 7930@unnumberedsec unnamed-faq-95 7931@example 7932@verbatim 7933To: Tielman Koekemoer <tielman@spi.co.za> 7934Subject: Re: Please help. 7935In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT. 7936Date: Thu, 08 Jul 1999 08:20:39 PDT 7937From: Vern Paxson <vern> 7938 7939> I was hoping you could help me with my problem. 7940> 7941> I tried compiling (gnu)flex on a Solaris 2.4 machine 7942> but when I ran make (after configure) I got an error. 7943> 7944> -------------------------------------------------------------- 7945> gcc -c -I. -I. -g -O parse.c 7946> ./flex -t -p ./scan.l >scan.c 7947> sh: ./flex: not found 7948> *** Error code 1 7949> make: Fatal error: Command failed for target `scan.c' 7950> ------------------------------------------------------------- 7951> 7952> What's strange to me is that I'm only 7953> trying to install flex now. I then edited the Makefile to 7954> and changed where it says "FLEX = flex" to "FLEX = lex" 7955> ( lex: the native Solaris one ) but then it complains about 7956> the "-p" option. Is there any way I can compile flex without 7957> using flex or lex? 7958> 7959> Thanks so much for your time. 7960 7961You managed to step on the bootstrap sequence, which first copies 7962initscan.c to scan.c in order to build flex. Try fetching a fresh 7963distribution from ftp.ee.lbl.gov. (Or you can first try removing 7964".bootstrap" and doing a make again.) 7965 7966 Vern 7967@end verbatim 7968@end example 7969 7970@c TODO: Evaluate this faq. 7971@node unnamed-faq-96 7972@unnumberedsec unnamed-faq-96 7973@example 7974@verbatim 7975To: Tielman Koekemoer <tielman@spi.co.za> 7976Subject: Re: Please help. 7977In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT. 7978Date: Fri, 09 Jul 1999 00:27:20 PDT 7979From: Vern Paxson <vern> 7980 7981> First I removed .bootstrap (and ran make) - no luck. I downloaded the 7982> software but I still have the same problem. Is there anything else I 7983> could try. 7984 7985Try: 7986 7987 cp initscan.c scan.c 7988 touch scan.c 7989 make scan.o 7990 7991If this last tries to first build scan.c from scan.l using ./flex, then 7992your "make" is broken, in which case compile scan.c to scan.o by hand. 7993 7994 Vern 7995@end verbatim 7996@end example 7997 7998@c TODO: Evaluate this faq. 7999@node unnamed-faq-97 8000@unnumberedsec unnamed-faq-97 8001@example 8002@verbatim 8003To: Sumanth Kamenani <skamenan@crl.nmsu.edu> 8004Subject: Re: Error 8005In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT. 8006Date: Tue, 20 Jul 1999 00:18:26 PDT 8007From: Vern Paxson <vern> 8008 8009> I am getting a compilation error. The error is given as "unknown symbol- yylex". 8010 8011The parser relies on calling yylex(), but you're instead using the C++ scanning 8012class, so you need to supply a yylex() "glue" function that calls an instance 8013scanner of the scanner (e.g., "scanner->yylex()"). 8014 8015 Vern 8016@end verbatim 8017@end example 8018 8019@c TODO: Evaluate this faq. 8020@node unnamed-faq-98 8021@unnumberedsec unnamed-faq-98 8022@example 8023@verbatim 8024To: daniel@synchrods.synchrods.COM (Daniel Senderowicz) 8025Subject: Re: lex 8026In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST. 8027Date: Tue, 23 Nov 1999 15:54:30 PST 8028From: Vern Paxson <vern> 8029 8030Well, your problem is the 8031 8032switch (yybgin-yysvec-1) { /* witchcraft */ 8033 8034at the beginning of lex rules. "witchcraft" == "non-portable". It's 8035assuming knowledge of the AT&T lex's internal variables. 8036 8037For flex, you can probably do the equivalent using a switch on YYSTATE. 8038 8039 Vern 8040@end verbatim 8041@end example 8042 8043@c TODO: Evaluate this faq. 8044@node unnamed-faq-99 8045@unnumberedsec unnamed-faq-99 8046@example 8047@verbatim 8048To: archow@hss.hns.com 8049Subject: Re: Regarding distribution of flex and yacc based grammars 8050In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530. 8051Date: Wed, 22 Dec 1999 01:56:24 PST 8052From: Vern Paxson <vern> 8053 8054> When we provide the customer with an object code distribution, is it 8055> necessary for us to provide source 8056> for the generated C files from flex and bison since they are generated by 8057> flex and bison ? 8058 8059For flex, no. I don't know what the current state of this is for bison. 8060 8061> Also, is there any requrirement for us to neccessarily provide source for 8062> the grammar files which are fed into flex and bison ? 8063 8064Again, for flex, no. 8065 8066See the file "COPYING" in the flex distribution for the legalese. 8067 8068 Vern 8069@end verbatim 8070@end example 8071 8072@c TODO: Evaluate this faq. 8073@node unnamed-faq-100 8074@unnumberedsec unnamed-faq-100 8075@example 8076@verbatim 8077To: Martin Gallwey <gallweym@hyperion.moe.ul.ie> 8078Subject: Re: Flex, and self referencing rules 8079In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST. 8080Date: Sat, 19 Feb 2000 18:33:16 PST 8081From: Vern Paxson <vern> 8082 8083> However, I do not use unput anywhere. I do use self-referencing 8084> rules like this: 8085> 8086> UnaryExpr ({UnionExpr})|("-"{UnaryExpr}) 8087 8088You can't do this - flex is *not* a parser like yacc (which does indeed 8089allow recursion), it is a scanner that's confined to regular expressions. 8090 8091 Vern 8092@end verbatim 8093@end example 8094 8095@c TODO: Evaluate this faq. 8096@node unnamed-faq-101 8097@unnumberedsec unnamed-faq-101 8098@example 8099@verbatim 8100To: slg3@lehigh.edu (SAMUEL L. GULDEN) 8101Subject: Re: Flex problem 8102In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST. 8103Date: Thu, 02 Mar 2000 23:00:46 PST 8104From: Vern Paxson <vern> 8105 8106If this is exactly your program: 8107 8108> digit [0-9] 8109> digits {digit}+ 8110> whitespace [ \t\n]+ 8111> 8112> %% 8113> "[" { printf("open_brac\n");} 8114> "]" { printf("close_brac\n");} 8115> "+" { printf("addop\n");} 8116> "*" { printf("multop\n");} 8117> {digits} { printf("NUMBER = %s\n", yytext);} 8118> whitespace ; 8119 8120then the problem is that the last rule needs to be "{whitespace}" ! 8121 8122 Vern 8123@end verbatim 8124@end example 8125 8126@node What is the difference between YYLEX_PARAM and YY_DECL? 8127@unnumberedsec What is the difference between YYLEX_PARAM and YY_DECL? 8128 8129YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to pass extra 8130params when it calls yylex() from the parser. 8131 8132YY_DECL is the Flex declaration of yylex. The default is similar to this: 8133 8134@example 8135@verbatim 8136#define int yy_lex () 8137@end verbatim 8138@end example 8139 8140 8141@node Why do I get "conflicting types for yylex" error? 8142@unnumberedsec Why do I get "conflicting types for yylex" error? 8143 8144This is a compiler error regarding a generated Bison parser, not a Flex scanner. 8145It means you need a prototype of yylex() in the top of the Bison file. 8146Be sure the prototype matches YY_DECL. 8147 8148@node How do I access the values set in a Flex action from within a Bison action? 8149@unnumberedsec How do I access the values set in a Flex action from within a Bison action? 8150 8151With $1, $2, $3, etc. These are called "Semantic Values" in the Bison manual. 8152See @ref{Top, , , bison, the GNU Bison Manual}. 8153 8154@node Appendices, Indices, FAQ, Top 8155@appendix Appendices 8156 8157@menu 8158* Makefiles and Flex:: 8159* Bison Bridge:: 8160* M4 Dependency:: 8161* Common Patterns:: 8162@end menu 8163 8164@node Makefiles and Flex, Bison Bridge, Appendices, Appendices 8165@appendixsec Makefiles and Flex 8166 8167@cindex Makefile, syntax 8168 8169In this appendix, we provide tips for writing Makefiles to build your scanners. 8170 8171In a traditional build environment, we say that the @file{.c} files are the 8172sources, and the @file{.o} files are the intermediate files. When using 8173@code{flex}, however, the @file{.l} files are the sources, and the generated 8174@file{.c} files (along with the @file{.o} files) are the intermediate files. 8175This requires you to carefully plan your Makefile. 8176 8177Modern @command{make} programs understand that @file{foo.l} is intended to 8178generate @file{lex.yy.c} or @file{foo.c}, and will behave 8179accordingly@footnote{GNU @command{make} and GNU @command{automake} are two such 8180programs that provide implicit rules for flex-generated scanners.}@footnote{GNU @command{automake} 8181may generate code to execute flex in lex-compatible mode, or to stdout. If this is not what you want, 8182then you should provide an explicit rule in your Makefile.am}. The 8183following Makefile does not explicitly instruct @command{make} how to build 8184@file{foo.c} from @file{foo.l}. Instead, it relies on the implicit rules of the 8185@command{make} program to build the intermediate file, @file{scan.c}: 8186 8187@cindex Makefile, example of implicit rules 8188@example 8189@verbatim 8190 # Basic Makefile -- relies on implicit rules 8191 # Creates "myprogram" from "scan.l" and "myprogram.c" 8192 # 8193 LEX=flex 8194 myprogram: scan.o myprogram.o 8195 scan.o: scan.l 8196 8197@end verbatim 8198@end example 8199 8200 8201For simple cases, the above may be sufficient. For other cases, 8202you may have to explicitly instruct @command{make} how to build your scanner. 8203The following is an example of a Makefile containing explicit rules: 8204 8205@cindex Makefile, explicit example 8206@example 8207@verbatim 8208 # Basic Makefile -- provides explicit rules 8209 # Creates "myprogram" from "scan.l" and "myprogram.c" 8210 # 8211 LEX=flex 8212 myprogram: scan.o myprogram.o 8213 $(CC) -o $@ $(LDFLAGS) $^ 8214 8215 myprogram.o: myprogram.c 8216 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ 8217 8218 scan.o: scan.c 8219 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ 8220 8221 scan.c: scan.l 8222 $(LEX) $(LFLAGS) -o $@ $^ 8223 8224 clean: 8225 $(RM) *.o scan.c 8226 8227@end verbatim 8228@end example 8229 8230Notice in the above example that @file{scan.c} is in the @code{clean} target. 8231This is because we consider the file @file{scan.c} to be an intermediate file. 8232 8233Finally, we provide a realistic example of a @code{flex} scanner used with a 8234@code{bison} parser@footnote{This example also applies to yacc parsers.}. 8235There is a tricky problem we have to deal with. Since a @code{flex} scanner 8236will typically include a header file (e.g., @file{y.tab.h}) generated by the 8237parser, we need to be sure that the header file is generated BEFORE the scanner 8238is compiled. We handle this case in the following example: 8239 8240@example 8241@verbatim 8242 # Makefile example -- scanner and parser. 8243 # Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c" 8244 # 8245 LEX = flex 8246 YACC = bison -y 8247 YFLAGS = -d 8248 objects = scan.o parse.o myprogram.o 8249 8250 myprogram: $(objects) 8251 scan.o: scan.l parse.c 8252 parse.o: parse.y 8253 myprogram.o: myprogram.c 8254 8255@end verbatim 8256@end example 8257 8258In the above example, notice the line, 8259 8260@example 8261@verbatim 8262 scan.o: scan.l parse.c 8263@end verbatim 8264@end example 8265 8266, which lists the file @file{parse.c} (the generated parser) as a dependency of 8267@file{scan.o}. We want to ensure that the parser is created before the scanner 8268is compiled, and the above line seems to do the trick. Feel free to experiment 8269with your specific implementation of @command{make}. 8270 8271 8272For more details on writing Makefiles, see @ref{Top, , , make, The 8273GNU Make Manual}. 8274 8275@node Bison Bridge, M4 Dependency, Makefiles and Flex, Appendices 8276@section C Scanners with Bison Parsers 8277 8278@cindex bison, bridging with flex 8279@vindex yylval 8280@vindex yylloc 8281@tindex YYLTYPE 8282@tindex YYSTYPE 8283 8284This section describes the @code{flex} features useful when integrating 8285@code{flex} with @code{GNU bison}@footnote{The features described here are 8286purely optional, and are by no means the only way to use flex with bison. 8287We merely provide some glue to ease development of your parser-scanner pair.}. 8288Skip this section if you are not using 8289@code{bison} with your scanner. Here we discuss only the @code{flex} 8290half of the @code{flex} and @code{bison} pair. We do not discuss 8291@code{bison} in any detail. For more information about generating 8292@code{bison} parsers, see @ref{Top, , , bison, the GNU Bison Manual}. 8293 8294A compatible @code{bison} scanner is generated by declaring @samp{%option 8295bison-bridge} or by supplying @samp{--bison-bridge} when invoking @code{flex} 8296from the command line. This instructs @code{flex} that the macro 8297@code{yylval} may be used. The data type for 8298@code{yylval}, @code{YYSTYPE}, 8299is typically defined in a header file, included in section 1 of the 8300@code{flex} input file. For a list of functions and macros 8301available, @xref{bison-functions}. 8302 8303The declaration of yylex becomes, 8304 8305@findex yylex (reentrant version) 8306@example 8307@verbatim 8308 int yylex ( YYSTYPE * lvalp, yyscan_t scanner ); 8309@end verbatim 8310@end example 8311 8312If @code{%option bison-locations} is specified, then the declaration 8313becomes, 8314 8315@findex yylex (reentrant version) 8316@example 8317@verbatim 8318 int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner ); 8319@end verbatim 8320@end example 8321 8322Note that the macros @code{yylval} and @code{yylloc} evaluate to pointers. 8323Support for @code{yylloc} is optional in @code{bison}, so it is optional in 8324@code{flex} as well. The following is an example of a @code{flex} scanner that 8325is compatible with @code{bison}. 8326 8327@cindex bison, scanner to be called from bison 8328@example 8329@verbatim 8330 /* Scanner for "C" assignment statements... sort of. */ 8331 %{ 8332 #include "y.tab.h" /* Generated by bison. */ 8333 %} 8334 8335 %option bison-bridge bison-locations 8336 % 8337 8338 [[:digit:]]+ { yylval->num = atoi(yytext); return NUMBER;} 8339 [[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;} 8340 "="|";" { return yytext[0];} 8341 . {} 8342 % 8343@end verbatim 8344@end example 8345 8346As you can see, there really is no magic here. We just use 8347@code{yylval} as we would any other variable. The data type of 8348@code{yylval} is generated by @code{bison}, and included in the file 8349@file{y.tab.h}. Here is the corresponding @code{bison} parser: 8350 8351@cindex bison, parser 8352@example 8353@verbatim 8354 /* Parser to convert "C" assignments to lisp. */ 8355 %{ 8356 /* Pass the argument to yyparse through to yylex. */ 8357 #define YYPARSE_PARAM scanner 8358 #define YYLEX_PARAM scanner 8359 %} 8360 %locations 8361 %pure_parser 8362 %union { 8363 int num; 8364 char* str; 8365 } 8366 %token <str> STRING 8367 %token <num> NUMBER 8368 %% 8369 assignment: 8370 STRING '=' NUMBER ';' { 8371 printf( "(setf %s %d)", $1, $3 ); 8372 } 8373 ; 8374@end verbatim 8375@end example 8376 8377@node M4 Dependency, Common Patterns, Bison Bridge, Appendices 8378@section M4 Dependency 8379@cindex m4 8380The macro processor @code{m4}@footnote{The use of m4 is subject to change in 8381future revisions of flex. It is not part of the public API of flex. Do not depend on it.} 8382must be installed wherever flex is installed. 8383@code{flex} invokes @samp{m4}, found by searching the directories in the 8384@code{PATH} environment variable. Any code you place in section 1 or in the 8385actions will be sent through m4. Please follow these rules to protect your 8386code from unwanted @code{m4} processing. 8387 8388@itemize 8389 8390@item Do not use symbols that begin with, @samp{m4_}, such as, @samp{m4_define}, 8391or @samp{m4_include}, since those are reserved for @code{m4} macro names. If for 8392some reason you need m4_ as a prefix, use a preprocessor #define to get your 8393symbol past m4 unmangled. 8394 8395@item Do not use the strings @samp{[[} or @samp{]]} anywhere in your code. The 8396former is not valid in C, except within comments and strings, but the latter is valid in 8397code such as @code{x[y[z]]}. The solution is simple. To get the literal string 8398@code{"]]"}, use @code{"]""]"}. To get the array notation @code{x[y[z]]}, 8399use @code{x[y[z] ]}. Flex will attempt to detect these sequences in user code, and 8400escape them. However, it's best to avoid this complexity where possible, by 8401removing such sequences from your code. 8402 8403@end itemize 8404 8405@code{m4} is only required at the time you run @code{flex}. The generated 8406scanner is ordinary C or C++, and does @emph{not} require @code{m4}. 8407 8408@node Common Patterns, ,M4 Dependency, Appendices 8409@section Common Patterns 8410@cindex patterns, common 8411 8412This appendix provides examples of common regular expressions you might use 8413in your scanner. 8414 8415@menu 8416* Numbers:: 8417* Identifiers:: 8418* Quoted Constructs:: 8419* Addresses:: 8420@end menu 8421 8422 8423@node Numbers, Identifiers, ,Common Patterns 8424@subsection Numbers 8425 8426@table @asis 8427 8428@item C99 decimal constant 8429@code{([[:digit:]]@{-@}[0])[[:digit:]]*} 8430 8431@item C99 hexadecimal constant 8432@code{0[xX][[:xdigit:]]+} 8433 8434@item C99 octal constant 8435@code{0[01234567]*} 8436 8437@item C99 floating point constant 8438@verbatim 8439 {dseq} ([[:digit:]]+) 8440 {dseq_opt} ([[:digit:]]*) 8441 {frac} (({dseq_opt}"."{dseq})|{dseq}".") 8442 {exp} ([eE][+-]?{dseq}) 8443 {exp_opt} ({exp}?) 8444 {fsuff} [flFL] 8445 {fsuff_opt} ({fsuff}?) 8446 {hpref} (0[xX]) 8447 {hdseq} ([[:xdigit:]]+) 8448 {hdseq_opt} ([[:xdigit:]]*) 8449 {hfrac} (({hdseq_opt}"."{hdseq})|({hdseq}".")) 8450 {bexp} ([pP][+-]?{dseq}) 8451 {dfc} (({frac}{exp_opt}{fsuff_opt})|({dseq}{exp}{fsuff_opt})) 8452 {hfc} (({hpref}{hfrac}{bexp}{fsuff_opt})|({hpref}{hdseq}{bexp}{fsuff_opt})) 8453 8454 {c99_floating_point_constant} ({dfc}|{hfc}) 8455@end verbatim 8456 8457See C99 section 6.4.4.2 for the gory details. 8458 8459@end table 8460 8461@node Identifiers, Quoted Constructs, Numbers, Common Patterns 8462@subsection Identifiers 8463 8464@table @asis 8465 8466@item C99 Identifier 8467@verbatim 8468ucn ((\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8}))) 8469nondigit [_[:alpha:]] 8470c99_id ([_[:alpha:]]|{ucn})([_[:alnum:]]|{ucn})* 8471@end verbatim 8472 8473Technically, the above pattern does not encompass all possible C99 identifiers, since C99 allows for 8474"implementation-defined" characters. In practice, C compilers follow the above pattern, with the 8475addition of the @samp{$} character. 8476 8477@item UTF-8 Encoded Unicode Code Point 8478@verbatim 8479[\x09\x0A\x0D\x20-\x7E]|[\xC2-\xDF][\x80-\xBF]|\xE0[\xA0-\xBF][\x80-\xBF]|[\xE1-\xEC\xEE\xEF]([\x80-\xBF]{2})|\xED[\x80-\x9F][\x80-\xBF]|\xF0[\x90-\xBF]([\x80-\xBF]{2})|[\xF1-\xF3]([\x80-\xBF]{3})|\xF4[\x80-\x8F]([\x80-\xBF]{2}) 8480@end verbatim 8481 8482@end table 8483 8484@node Quoted Constructs, Addresses, Identifiers, Common Patterns 8485@subsection Quoted Constructs 8486 8487@table @asis 8488@item C99 String Literal 8489@code{L?\"([^\"\\\n]|(\\['\"?\\abfnrtv])|(\\([0123456]@{1,3@}))|(\\x[[:xdigit:]]+)|(\\u([[:xdigit:]]@{4@}))|(\\U([[:xdigit:]]@{8@})))*\"} 8490 8491@item C99 Comment 8492@code{("/*"([^*]|"*"[^/])*"*/")|("/"(\\\n)*"/"[^\n]*)} 8493 8494Note that in C99, a @samp{//}-style comment may be split across lines, and, contrary to popular belief, 8495does not include the trailing @samp{\n} character. 8496 8497A better way to scan @samp{/* */} comments is by line, rather than matching 8498possibly huge comments all at once. This will allow you to scan comments of 8499unlimited length, as long as line breaks appear at sane intervals. This is also 8500more efficient when used with automatic line number processing. @xref{option-yylineno}. 8501 8502@verbatim 8503<INITIAL>{ 8504 "/*" BEGIN(COMMENT); 8505} 8506<COMMENT>{ 8507 "*/" BEGIN(0); 8508 [^*\n]+ ; 8509 "*"[^/] ; 8510 \n ; 8511} 8512@end verbatim 8513 8514@end table 8515 8516@node Addresses, ,Quoted Constructs, Common Patterns 8517@subsection Addresses 8518 8519@table @asis 8520 8521@item IPv4 Address 8522@verbatim 8523dec-octet [0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5] 8524IPv4address {dec-octet}\.{dec-octet}\.{dec-octet}\.{dec-octet} 8525@end verbatim 8526 8527@item IPv6 Address 8528@verbatim 8529h16 [0-9A-Fa-f]{1,4} 8530ls32 {h16}:{h16}|{IPv4address} 8531IPv6address ({h16}:){6}{ls32}| 8532 ::({h16}:){5}{ls32}| 8533 ({h16})?::({h16}:){4}{ls32}| 8534 (({h16}:){0,1}{h16})?::({h16}:){3}{ls32}| 8535 (({h16}:){0,2}{h16})?::({h16}:){2}{ls32}| 8536 (({h16}:){0,3}{h16})?::{h16}:{ls32}| 8537 (({h16}:){0,4}{h16})?::{ls32}| 8538 (({h16}:){0,5}{h16})?::{h16}| 8539 (({h16}:){0,6}{h16})?:: 8540@end verbatim 8541 8542See @uref{http://www.ietf.org/rfc/rfc2373.txt, RFC 2373} for details. 8543Note that you have to fold the definition of @code{IPv6address} into one 8544line and that it also matches the ``unspecified address'' ``::''. 8545 8546@item URI 8547@code{(([^:/?#]+):)?("//"([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?} 8548 8549This pattern is nearly useless, since it allows just about any character 8550to appear in a URI, including spaces and control characters. See 8551@uref{http://www.ietf.org/rfc/rfc2396.txt, RFC 2396} for details. 8552 8553@end table 8554 8555 8556@node Indices, , Appendices, Top 8557@unnumbered Indices 8558 8559@menu 8560* Concept Index:: 8561* Index of Functions and Macros:: 8562* Index of Variables:: 8563* Index of Data Types:: 8564* Index of Hooks:: 8565* Index of Scanner Options:: 8566@end menu 8567 8568@node Concept Index, Index of Functions and Macros, Indices, Indices 8569@unnumberedsec Concept Index 8570 8571@printindex cp 8572 8573@node Index of Functions and Macros, Index of Variables, Concept Index, Indices 8574@unnumberedsec Index of Functions and Macros 8575 8576This is an index of functions and preprocessor macros that look like functions. 8577For macros that expand to variables or constants, see @ref{Index of Variables}. 8578 8579@printindex fn 8580 8581@node Index of Variables, Index of Data Types, Index of Functions and Macros, Indices 8582@unnumberedsec Index of Variables 8583 8584This is an index of variables, constants, and preprocessor macros 8585that expand to variables or constants. 8586 8587@printindex vr 8588 8589@node Index of Data Types, Index of Hooks, Index of Variables, Indices 8590@unnumberedsec Index of Data Types 8591@printindex tp 8592 8593@node Index of Hooks, Index of Scanner Options, Index of Data Types, Indices 8594@unnumberedsec Index of Hooks 8595 8596This is an index of "hooks" that the user may define. These hooks typically correspond 8597to specific locations in the generated scanner, and may be used to insert arbitrary code. 8598 8599@printindex hk 8600 8601@node Index of Scanner Options, , Index of Hooks, Indices 8602@unnumberedsec Index of Scanner Options 8603 8604@printindex op 8605 8606@c A vim script to name the faq entries. delete this when faqs are no longer 8607@c named "unnamed-faq-XXX". 8608@c 8609@c fu! Faq2 () range abort 8610@c let @r=input("Rename to: ") 8611@c exe "%s/" . @w . "/" . @r . "/g" 8612@c normal 'f 8613@c endf 8614@c nnoremap <F5> 1G/@node\s\+unnamed-faq-\d\+<cr>mfww"wy5ezt:call Faq2()<cr> 8615 8616@bye 8617