1This is flex.info, produced by makeinfo version 6.0 from flex.texi. 2 3The flex manual is placed under the same licensing conditions as the 4rest of flex: 5 6 Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012 The Flex 7Project. 8 9 Copyright (C) 1990, 1997 The Regents of the University of California. 10All rights reserved. 11 12 This code is derived from software contributed to Berkeley by Vern 13Paxson. 14 15 The United States Government has rights in this work pursuant to 16contract no. DE-AC03-76SF00098 between the United States Department of 17Energy and the University of California. 18 19 Redistribution and use in source and binary forms, with or without 20modification, are permitted provided that the following conditions are 21met: 22 23 1. Redistributions of source code must retain the above copyright 24 notice, this list of conditions and the following disclaimer. 25 26 2. Redistributions in binary form must reproduce the above copyright 27 notice, this list of conditions and the following disclaimer in the 28 documentation and/or other materials provided with the 29 distribution. 30 31 Neither the name of the University nor the names of its contributors 32may be used to endorse or promote products derived from this software 33without specific prior written permission. 34 35 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED 36WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF 37MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 38INFO-DIR-SECTION Programming 39START-INFO-DIR-ENTRY 40* flex: (flex). Fast lexical analyzer generator (lex replacement). 41END-INFO-DIR-ENTRY 42 43 44File: flex.info, Node: Top, Next: Copyright, Prev: (dir), Up: (dir) 45 46flex 47**** 48 49This manual describes 'flex', a tool for generating programs that 50perform pattern-matching on text. The manual includes both tutorial and 51reference sections. 52 53 This edition of 'The flex Manual' documents 'flex' version 2.6.0. It 54was last updated on 10 November 2015. 55 56 This manual was written by Vern Paxson, Will Estes and John Millaway. 57 58* Menu: 59 60* Copyright:: 61* Reporting Bugs:: 62* Introduction:: 63* Simple Examples:: 64* Format:: 65* Patterns:: 66* Matching:: 67* Actions:: 68* Generated Scanner:: 69* Start Conditions:: 70* Multiple Input Buffers:: 71* EOF:: 72* Misc Macros:: 73* User Values:: 74* Yacc:: 75* Scanner Options:: 76* Performance:: 77* Cxx:: 78* Reentrant:: 79* Lex and Posix:: 80* Memory Management:: 81* Serialized Tables:: 82* Diagnostics:: 83* Limitations:: 84* Bibliography:: 85* FAQ:: 86* Appendices:: 87* Indices:: 88 89 -- The Detailed Node Listing -- 90 91Format of the Input File 92 93* Definitions Section:: 94* Rules Section:: 95* User Code Section:: 96* Comments in the Input:: 97 98Scanner Options 99 100* Options for Specifying Filenames:: 101* Options Affecting Scanner Behavior:: 102* Code-Level And API Options:: 103* Options for Scanner Speed and Size:: 104* Debugging Options:: 105* Miscellaneous Options:: 106 107Reentrant C Scanners 108 109* Reentrant Uses:: 110* Reentrant Overview:: 111* Reentrant Example:: 112* Reentrant Detail:: 113* Reentrant Functions:: 114 115The Reentrant API in Detail 116 117* Specify Reentrant:: 118* Extra Reentrant Argument:: 119* Global Replacement:: 120* Init and Destroy Functions:: 121* Accessor Methods:: 122* Extra Data:: 123* About yyscan_t:: 124 125Memory Management 126 127* The Default Memory Management:: 128* Overriding The Default Memory Management:: 129* A Note About yytext And Memory:: 130 131Serialized Tables 132 133* Creating Serialized Tables:: 134* Loading and Unloading Serialized Tables:: 135* Tables File Format:: 136 137FAQ 138 139* When was flex born?:: 140* How do I expand backslash-escape sequences in C-style quoted strings?:: 141* Why do flex scanners call fileno if it is not ANSI compatible?:: 142* Does flex support recursive pattern definitions?:: 143* How do I skip huge chunks of input (tens of megabytes) while using flex?:: 144* Flex is not matching my patterns in the same order that I defined them.:: 145* My actions are executing out of order or sometimes not at all.:: 146* How can I have multiple input sources feed into the same scanner at the same time?:: 147* Can I build nested parsers that work with the same input file?:: 148* How can I match text only at the end of a file?:: 149* How can I make REJECT cascade across start condition boundaries?:: 150* Why cant I use fast or full tables with interactive mode?:: 151* How much faster is -F or -f than -C?:: 152* If I have a simple grammar cant I just parse it with flex?:: 153* Why doesn't yyrestart() set the start state back to INITIAL?:: 154* How can I match C-style comments?:: 155* The period isn't working the way I expected.:: 156* Can I get the flex manual in another format?:: 157* Does there exist a "faster" NDFA->DFA algorithm?:: 158* How does flex compile the DFA so quickly?:: 159* How can I use more than 8192 rules?:: 160* How do I abandon a file in the middle of a scan and switch to a new file?:: 161* How do I execute code only during initialization (only before the first scan)?:: 162* How do I execute code at termination?:: 163* Where else can I find help?:: 164* Can I include comments in the "rules" section of the file?:: 165* I get an error about undefined yywrap().:: 166* How can I change the matching pattern at run time?:: 167* How can I expand macros in the input?:: 168* How can I build a two-pass scanner?:: 169* How do I match any string not matched in the preceding rules?:: 170* I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: 171* Is there a way to make flex treat NULL like a regular character?:: 172* Whenever flex can not match the input it says "flex scanner jammed".:: 173* Why doesn't flex have non-greedy operators like perl does?:: 174* Memory leak - 16386 bytes allocated by malloc.:: 175* How do I track the byte offset for lseek()?:: 176* How do I use my own I/O classes in a C++ scanner?:: 177* How do I skip as many chars as possible?:: 178* deleteme00:: 179* Are certain equivalent patterns faster than others?:: 180* Is backing up a big deal?:: 181* Can I fake multi-byte character support?:: 182* deleteme01:: 183* Can you discuss some flex internals?:: 184* unput() messes up yy_at_bol:: 185* The | operator is not doing what I want:: 186* Why can't flex understand this variable trailing context pattern?:: 187* The ^ operator isn't working:: 188* Trailing context is getting confused with trailing optional patterns:: 189* Is flex GNU or not?:: 190* ERASEME53:: 191* I need to scan if-then-else blocks and while loops:: 192* ERASEME55:: 193* ERASEME56:: 194* ERASEME57:: 195* Is there a repository for flex scanners?:: 196* How can I conditionally compile or preprocess my flex input file?:: 197* Where can I find grammars for lex and yacc?:: 198* I get an end-of-buffer message for each character scanned.:: 199* unnamed-faq-62:: 200* unnamed-faq-63:: 201* unnamed-faq-64:: 202* unnamed-faq-65:: 203* unnamed-faq-66:: 204* unnamed-faq-67:: 205* unnamed-faq-68:: 206* unnamed-faq-69:: 207* unnamed-faq-70:: 208* unnamed-faq-71:: 209* unnamed-faq-72:: 210* unnamed-faq-73:: 211* unnamed-faq-74:: 212* unnamed-faq-75:: 213* unnamed-faq-76:: 214* unnamed-faq-77:: 215* unnamed-faq-78:: 216* unnamed-faq-79:: 217* unnamed-faq-80:: 218* unnamed-faq-81:: 219* unnamed-faq-82:: 220* unnamed-faq-83:: 221* unnamed-faq-84:: 222* unnamed-faq-85:: 223* unnamed-faq-86:: 224* unnamed-faq-87:: 225* unnamed-faq-88:: 226* unnamed-faq-90:: 227* unnamed-faq-91:: 228* unnamed-faq-92:: 229* unnamed-faq-93:: 230* unnamed-faq-94:: 231* unnamed-faq-95:: 232* unnamed-faq-96:: 233* unnamed-faq-97:: 234* unnamed-faq-98:: 235* unnamed-faq-99:: 236* unnamed-faq-100:: 237* unnamed-faq-101:: 238* What is the difference between YYLEX_PARAM and YY_DECL?:: 239* Why do I get "conflicting types for yylex" error?:: 240* How do I access the values set in a Flex action from within a Bison action?:: 241 242Appendices 243 244* Makefiles and Flex:: 245* Bison Bridge:: 246* M4 Dependency:: 247* Common Patterns:: 248 249Indices 250 251* Concept Index:: 252* Index of Functions and Macros:: 253* Index of Variables:: 254* Index of Data Types:: 255* Index of Hooks:: 256* Index of Scanner Options:: 257 258 259 260File: flex.info, Node: Copyright, Next: Reporting Bugs, Prev: Top, Up: Top 261 2621 Copyright 263*********** 264 265The flex manual is placed under the same licensing conditions as the 266rest of flex: 267 268 Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012 The Flex 269Project. 270 271 Copyright (C) 1990, 1997 The Regents of the University of California. 272All rights reserved. 273 274 This code is derived from software contributed to Berkeley by Vern 275Paxson. 276 277 The United States Government has rights in this work pursuant to 278contract no. DE-AC03-76SF00098 between the United States Department of 279Energy and the University of California. 280 281 Redistribution and use in source and binary forms, with or without 282modification, are permitted provided that the following conditions are 283met: 284 285 1. Redistributions of source code must retain the above copyright 286 notice, this list of conditions and the following disclaimer. 287 288 2. Redistributions in binary form must reproduce the above copyright 289 notice, this list of conditions and the following disclaimer in the 290 documentation and/or other materials provided with the 291 distribution. 292 293 Neither the name of the University nor the names of its contributors 294may be used to endorse or promote products derived from this software 295without specific prior written permission. 296 297 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED 298WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF 299MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 300 301 302File: flex.info, Node: Reporting Bugs, Next: Introduction, Prev: Copyright, Up: Top 303 3042 Reporting Bugs 305**************** 306 307If you find a bug in 'flex', please report it using the SourceForge Bug 308Tracking facilities which can be found on flex's SourceForge Page 309(http://sourceforge.net/projects/flex). 310 311 312File: flex.info, Node: Introduction, Next: Simple Examples, Prev: Reporting Bugs, Up: Top 313 3143 Introduction 315************** 316 317'flex' is a tool for generating "scanners". A scanner is a program 318which recognizes lexical patterns in text. The 'flex' program reads the 319given input files, or its standard input if no file names are given, for 320a description of a scanner to generate. The description is in the form 321of pairs of regular expressions and C code, called "rules". 'flex' 322generates as output a C source file, 'lex.yy.c' by default, which 323defines a routine 'yylex()'. This file can be compiled and linked with 324the flex runtime library to produce an executable. When the executable 325is run, it analyzes its input for occurrences of the regular 326expressions. Whenever it finds one, it executes the corresponding C 327code. 328 329 330File: flex.info, Node: Simple Examples, Next: Format, Prev: Introduction, Up: Top 331 3324 Some Simple Examples 333********************** 334 335First some simple examples to get the flavor of how one uses 'flex'. 336 337 The following 'flex' input specifies a scanner which, when it 338encounters the string 'username' will replace it with the user's login 339name: 340 341 %% 342 username printf( "%s", getlogin() ); 343 344 By default, any text not matched by a 'flex' scanner is copied to the 345output, so the net effect of this scanner is to copy its input file to 346its output with each occurrence of 'username' expanded. In this input, 347there is just one rule. 'username' is the "pattern" and the 'printf' is 348the "action". The '%%' symbol marks the beginning of the rules. 349 350 Here's another simple example: 351 352 int num_lines = 0, num_chars = 0; 353 354 %% 355 \n ++num_lines; ++num_chars; 356 . ++num_chars; 357 358 %% 359 360 int main() 361 { 362 yylex(); 363 printf( "# of lines = %d, # of chars = %d\n", 364 num_lines, num_chars ); 365 } 366 367 This scanner counts the number of characters and the number of lines 368in its input. It produces no output other than the final report on the 369character and line counts. The first line declares two globals, 370'num_lines' and 'num_chars', which are accessible both inside 'yylex()' 371and in the 'main()' routine declared after the second '%%'. There are 372two rules, one which matches a newline ('\n') and increments both the 373line count and the character count, and one which matches any character 374other than a newline (indicated by the '.' regular expression). 375 376 A somewhat more complicated example: 377 378 /* scanner for a toy Pascal-like language */ 379 380 %{ 381 /* need this for the call to atof() below */ 382 #include <math.h> 383 %} 384 385 DIGIT [0-9] 386 ID [a-z][a-z0-9]* 387 388 %% 389 390 {DIGIT}+ { 391 printf( "An integer: %s (%d)\n", yytext, 392 atoi( yytext ) ); 393 } 394 395 {DIGIT}+"."{DIGIT}* { 396 printf( "A float: %s (%g)\n", yytext, 397 atof( yytext ) ); 398 } 399 400 if|then|begin|end|procedure|function { 401 printf( "A keyword: %s\n", yytext ); 402 } 403 404 {ID} printf( "An identifier: %s\n", yytext ); 405 406 "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext ); 407 408 "{"[\^{}}\n]*"}" /* eat up one-line comments */ 409 410 [ \t\n]+ /* eat up whitespace */ 411 412 . printf( "Unrecognized character: %s\n", yytext ); 413 414 %% 415 416 int main( int argc, char **argv ) 417 { 418 ++argv, --argc; /* skip over program name */ 419 if ( argc > 0 ) 420 yyin = fopen( argv[0], "r" ); 421 else 422 yyin = stdin; 423 424 yylex(); 425 } 426 427 This is the beginnings of a simple scanner for a language like 428Pascal. It identifies different types of "tokens" and reports on what 429it has seen. 430 431 The details of this example will be explained in the following 432sections. 433 434 435File: flex.info, Node: Format, Next: Patterns, Prev: Simple Examples, Up: Top 436 4375 Format of the Input File 438************************** 439 440The 'flex' input file consists of three sections, separated by a line 441containing only '%%'. 442 443 definitions 444 %% 445 rules 446 %% 447 user code 448 449* Menu: 450 451* Definitions Section:: 452* Rules Section:: 453* User Code Section:: 454* Comments in the Input:: 455 456 457File: flex.info, Node: Definitions Section, Next: Rules Section, Prev: Format, Up: Format 458 4595.1 Format of the Definitions Section 460===================================== 461 462The "definitions section" contains declarations of simple "name" 463definitions to simplify the scanner specification, and declarations of 464"start conditions", which are explained in a later section. 465 466 Name definitions have the form: 467 468 name definition 469 470 The 'name' is a word beginning with a letter or an underscore ('_') 471followed by zero or more letters, digits, '_', or '-' (dash). The 472definition is taken to begin at the first non-whitespace character 473following the name and continuing to the end of the line. The 474definition can subsequently be referred to using '{name}', which will 475expand to '(definition)'. For example, 476 477 DIGIT [0-9] 478 ID [a-z][a-z0-9]* 479 480 Defines 'DIGIT' to be a regular expression which matches a single 481digit, and 'ID' to be a regular expression which matches a letter 482followed by zero-or-more letters-or-digits. A subsequent reference to 483 484 {DIGIT}+"."{DIGIT}* 485 486 is identical to 487 488 ([0-9])+"."([0-9])* 489 490 and matches one-or-more digits followed by a '.' followed by 491zero-or-more digits. 492 493 An unindented comment (i.e., a line beginning with '/*') is copied 494verbatim to the output up to the next '*/'. 495 496 Any _indented_ text or text enclosed in '%{' and '%}' is also copied 497verbatim to the output (with the %{ and %} symbols removed). The %{ and 498%} symbols must appear unindented on lines by themselves. 499 500 A '%top' block is similar to a '%{' ... '%}' block, except that the 501code in a '%top' block is relocated to the _top_ of the generated file, 502before any flex definitions (1). The '%top' block is useful when you 503want certain preprocessor macros to be defined or certain files to be 504included before the generated code. The single characters, '{' and '}' 505are used to delimit the '%top' block, as show in the example below: 506 507 %top{ 508 /* This code goes at the "top" of the generated file. */ 509 #include <stdint.h> 510 #include <inttypes.h> 511 } 512 513 Multiple '%top' blocks are allowed, and their order is preserved. 514 515 ---------- Footnotes ---------- 516 517 (1) Actually, 'yyIN_HEADER' is defined before the '%top' block. 518 519 520File: flex.info, Node: Rules Section, Next: User Code Section, Prev: Definitions Section, Up: Format 521 5225.2 Format of the Rules Section 523=============================== 524 525The "rules" section of the 'flex' input contains a series of rules of 526the form: 527 528 pattern action 529 530 where the pattern must be unindented and the action must begin on the 531same line. *Note Patterns::, for a further description of patterns and 532actions. 533 534 In the rules section, any indented or %{ %} enclosed text appearing 535before the first rule may be used to declare variables which are local 536to the scanning routine and (after the declarations) code which is to be 537executed whenever the scanning routine is entered. Other indented or %{ 538%} text in the rule section is still copied to the output, but its 539meaning is not well-defined and it may well cause compile-time errors 540(this feature is present for POSIX compliance. *Note Lex and Posix::, 541for other such features). 542 543 Any _indented_ text or text enclosed in '%{' and '%}' is copied 544verbatim to the output (with the %{ and %} symbols removed). The %{ and 545%} symbols must appear unindented on lines by themselves. 546 547 548File: flex.info, Node: User Code Section, Next: Comments in the Input, Prev: Rules Section, Up: Format 549 5505.3 Format of the User Code Section 551=================================== 552 553The user code section is simply copied to 'lex.yy.c' verbatim. It is 554used for companion routines which call or are called by the scanner. 555The presence of this section is optional; if it is missing, the second 556'%%' in the input file may be skipped, too. 557 558 559File: flex.info, Node: Comments in the Input, Prev: User Code Section, Up: Format 560 5615.4 Comments in the Input 562========================= 563 564Flex supports C-style comments, that is, anything between '/*' and '*/' 565is considered a comment. Whenever flex encounters a comment, it copies 566the entire comment verbatim to the generated source code. Comments may 567appear just about anywhere, but with the following exceptions: 568 569 * Comments may not appear in the Rules Section wherever flex is 570 expecting a regular expression. This means comments may not appear 571 at the beginning of a line, or immediately following a list of 572 scanner states. 573 * Comments may not appear on an '%option' line in the Definitions 574 Section. 575 576 If you want to follow a simple rule, then always begin a comment on a 577new line, with one or more whitespace characters before the initial 578'/*'). This rule will work anywhere in the input file. 579 580 All the comments in the following example are valid: 581 582 %{ 583 /* code block */ 584 %} 585 586 /* Definitions Section */ 587 %x STATE_X 588 589 %% 590 /* Rules Section */ 591 ruleA /* after regex */ { /* code block */ } /* after code block */ 592 /* Rules Section (indented) */ 593 <STATE_X>{ 594 ruleC ECHO; 595 ruleD ECHO; 596 %{ 597 /* code block */ 598 %} 599 } 600 %% 601 /* User Code Section */ 602 603 604 605File: flex.info, Node: Patterns, Next: Matching, Prev: Format, Up: Top 606 6076 Patterns 608********** 609 610The patterns in the input (see *note Rules Section::) are written using 611an extended set of regular expressions. These are: 612 613'x' 614 match the character 'x' 615 616'.' 617 any character (byte) except newline 618 619'[xyz]' 620 a "character class"; in this case, the pattern matches either an 621 'x', a 'y', or a 'z' 622 623'[abj-oZ]' 624 a "character class" with a range in it; matches an 'a', a 'b', any 625 letter from 'j' through 'o', or a 'Z' 626 627'[^A-Z]' 628 a "negated character class", i.e., any character but those in the 629 class. In this case, any character EXCEPT an uppercase letter. 630 631'[^A-Z\n]' 632 any character EXCEPT an uppercase letter or a newline 633 634'[a-z]{-}[aeiou]' 635 the lowercase consonants 636 637'r*' 638 zero or more r's, where r is any regular expression 639 640'r+' 641 one or more r's 642 643'r?' 644 zero or one r's (that is, "an optional r") 645 646'r{2,5}' 647 anywhere from two to five r's 648 649'r{2,}' 650 two or more r's 651 652'r{4}' 653 exactly 4 r's 654 655'{name}' 656 the expansion of the 'name' definition (*note Format::). 657 658'"[xyz]\"foo"' 659 the literal string: '[xyz]"foo' 660 661'\X' 662 if X is 'a', 'b', 'f', 'n', 'r', 't', or 'v', then the ANSI-C 663 interpretation of '\x'. Otherwise, a literal 'X' (used to escape 664 operators such as '*') 665 666'\0' 667 a NUL character (ASCII code 0) 668 669'\123' 670 the character with octal value 123 671 672'\x2a' 673 the character with hexadecimal value 2a 674 675'(r)' 676 match an 'r'; parentheses are used to override precedence (see 677 below) 678 679'(?r-s:pattern)' 680 apply option 'r' and omit option 's' while interpreting pattern. 681 Options may be zero or more of the characters 'i', 's', or 'x'. 682 683 'i' means case-insensitive. '-i' means case-sensitive. 684 685 's' alters the meaning of the '.' syntax to match any single byte 686 whatsoever. '-s' alters the meaning of '.' to match any byte 687 except '\n'. 688 689 'x' ignores comments and whitespace in patterns. Whitespace is 690 ignored unless it is backslash-escaped, contained within '""'s, or 691 appears inside a character class. 692 693 The following are all valid: 694 695 (?:foo) same as (foo) 696 (?i:ab7) same as ([aA][bB]7) 697 (?-i:ab) same as (ab) 698 (?s:.) same as [\x00-\xFF] 699 (?-s:.) same as [^\n] 700 (?ix-s: a . b) same as ([Aa][^\n][bB]) 701 (?x:a b) same as ("ab") 702 (?x:a\ b) same as ("a b") 703 (?x:a" "b) same as ("a b") 704 (?x:a[ ]b) same as ("a b") 705 (?x:a 706 /* comment */ 707 b 708 c) same as (abc) 709 710'(?# comment )' 711 omit everything within '()'. The first ')' character encountered 712 ends the pattern. It is not possible to for the comment to contain 713 a ')' character. The comment may span lines. 714 715'rs' 716 the regular expression 'r' followed by the regular expression 's'; 717 called "concatenation" 718 719'r|s' 720 either an 'r' or an 's' 721 722'r/s' 723 an 'r' but only if it is followed by an 's'. The text matched by 724 's' is included when determining whether this rule is the longest 725 match, but is then returned to the input before the action is 726 executed. So the action only sees the text matched by 'r'. This 727 type of pattern is called "trailing context". (There are some 728 combinations of 'r/s' that flex cannot match correctly. *Note 729 Limitations::, regarding dangerous trailing context.) 730 731'^r' 732 an 'r', but only at the beginning of a line (i.e., when just 733 starting to scan, or right after a newline has been scanned). 734 735'r$' 736 an 'r', but only at the end of a line (i.e., just before a 737 newline). Equivalent to 'r/\n'. 738 739 Note that 'flex''s notion of "newline" is exactly whatever the C 740 compiler used to compile 'flex' interprets '\n' as; in particular, 741 on some DOS systems you must either filter out '\r's in the input 742 yourself, or explicitly use 'r/\r\n' for 'r$'. 743 744'<s>r' 745 an 'r', but only in start condition 's' (see *note Start 746 Conditions:: for discussion of start conditions). 747 748'<s1,s2,s3>r' 749 same, but in any of start conditions 's1', 's2', or 's3'. 750 751'<*>r' 752 an 'r' in any start condition, even an exclusive one. 753 754'<<EOF>>' 755 an end-of-file. 756 757'<s1,s2><<EOF>>' 758 an end-of-file when in start condition 's1' or 's2' 759 760 Note that inside of a character class, all regular expression 761operators lose their special meaning except escape ('\') and the 762character class operators, '-', ']]', and, at the beginning of the 763class, '^'. 764 765 The regular expressions listed above are grouped according to 766precedence, from highest precedence at the top to lowest at the bottom. 767Those grouped together have equal precedence (see special note on the 768precedence of the repeat operator, '{}', under the documentation for the 769'--posix' POSIX compliance option). For example, 770 771 foo|bar* 772 773 is the same as 774 775 (foo)|(ba(r*)) 776 777 since the '*' operator has higher precedence than concatenation, and 778concatenation higher than alternation ('|'). This pattern therefore 779matches _either_ the string 'foo' _or_ the string 'ba' followed by 780zero-or-more 'r''s. To match 'foo' or zero-or-more repetitions of the 781string 'bar', use: 782 783 foo|(bar)* 784 785 And to match a sequence of zero or more repetitions of 'foo' and 786'bar': 787 788 (foo|bar)* 789 790 In addition to characters and ranges of characters, character classes 791can also contain "character class expressions". These are expressions 792enclosed inside '[:' and ':]' delimiters (which themselves must appear 793between the '[' and ']' of the character class. Other elements may 794occur inside the character class, too). The valid expressions are: 795 796 [:alnum:] [:alpha:] [:blank:] 797 [:cntrl:] [:digit:] [:graph:] 798 [:lower:] [:print:] [:punct:] 799 [:space:] [:upper:] [:xdigit:] 800 801 These expressions all designate a set of characters equivalent to the 802corresponding standard C 'isXXX' function. For example, '[:alnum:]' 803designates those characters for which 'isalnum()' returns true - i.e., 804any alphabetic or numeric character. Some systems don't provide 805'isblank()', so flex defines '[:blank:]' as a blank or a tab. 806 807 For example, the following character classes are all equivalent: 808 809 [[:alnum:]] 810 [[:alpha:][:digit:]] 811 [[:alpha:][0-9]] 812 [a-zA-Z0-9] 813 814 A word of caution. Character classes are expanded immediately when 815seen in the 'flex' input. This means the character classes are 816sensitive to the locale in which 'flex' is executed, and the resulting 817scanner will not be sensitive to the runtime locale. This may or may 818not be desirable. 819 820 * If your scanner is case-insensitive (the '-i' flag), then 821 '[:upper:]' and '[:lower:]' are equivalent to '[:alpha:]'. 822 823 * Character classes with ranges, such as '[a-Z]', should be used with 824 caution in a case-insensitive scanner if the range spans upper or 825 lowercase characters. Flex does not know if you want to fold all 826 upper and lowercase characters together, or if you want the literal 827 numeric range specified (with no case folding). When in doubt, 828 flex will assume that you meant the literal numeric range, and will 829 issue a warning. The exception to this rule is a character range 830 such as '[a-z]' or '[S-W]' where it is obvious that you want 831 case-folding to occur. Here are some examples with the '-i' flag 832 enabled: 833 834 Range Result Literal Range Alternate Range 835 '[a-t]' ok '[a-tA-T]' 836 '[A-T]' ok '[a-tA-T]' 837 '[A-t]' ambiguous '[A-Z\[\\\]_`a-t]' '[a-tA-T]' 838 '[_-{]' ambiguous '[_`a-z{]' '[_`a-zA-Z{]' 839 '[@-C]' ambiguous '[@ABC]' '[@A-Z\[\\\]_`abc]' 840 841 * A negated character class such as the example '[^A-Z]' above _will_ 842 match a newline unless '\n' (or an equivalent escape sequence) is 843 one of the characters explicitly present in the negated character 844 class (e.g., '[^A-Z\n]'). This is unlike how many other regular 845 expression tools treat negated character classes, but unfortunately 846 the inconsistency is historically entrenched. Matching newlines 847 means that a pattern like '[^"]*' can match the entire input unless 848 there's another quote in the input. 849 850 Flex allows negation of character class expressions by prepending 851 '^' to the POSIX character class name. 852 853 [:^alnum:] [:^alpha:] [:^blank:] 854 [:^cntrl:] [:^digit:] [:^graph:] 855 [:^lower:] [:^print:] [:^punct:] 856 [:^space:] [:^upper:] [:^xdigit:] 857 858 Flex will issue a warning if the expressions '[:^upper:]' and 859 '[:^lower:]' appear in a case-insensitive scanner, since their 860 meaning is unclear. The current behavior is to skip them entirely, 861 but this may change without notice in future revisions of flex. 862 863 * 864 The '{-}' operator computes the difference of two character 865 classes. For example, '[a-c]{-}[b-z]' represents all the 866 characters in the class '[a-c]' that are not in the class '[b-z]' 867 (which in this case, is just the single character 'a'). The '{-}' 868 operator is left associative, so '[abc]{-}[b]{-}[c]' is the same as 869 '[a]'. Be careful not to accidentally create an empty set, which 870 will never match. 871 872 * 873 The '{+}' operator computes the union of two character classes. 874 For example, '[a-z]{+}[0-9]' is the same as '[a-z0-9]'. This 875 operator is useful when preceded by the result of a difference 876 operation, as in, '[[:alpha:]]{-}[[:lower:]]{+}[q]', which is 877 equivalent to '[A-Zq]' in the "C" locale. 878 879 * A rule can have at most one instance of trailing context (the '/' 880 operator or the '$' operator). The start condition, '^', and 881 '<<EOF>>' patterns can only occur at the beginning of a pattern, 882 and, as well as with '/' and '$', cannot be grouped inside 883 parentheses. A '^' which does not occur at the beginning of a rule 884 or a '$' which does not occur at the end of a rule loses its 885 special properties and is treated as a normal character. 886 887 * The following are invalid: 888 889 foo/bar$ 890 <sc1>foo<sc2>bar 891 892 Note that the first of these can be written 'foo/bar\n'. 893 894 * The following will result in '$' or '^' being treated as a normal 895 character: 896 897 foo|(bar$) 898 foo|^bar 899 900 If the desired meaning is a 'foo' or a 'bar'-followed-by-a-newline, 901 the following could be used (the special '|' action is explained 902 below, *note Actions::): 903 904 foo | 905 bar$ /* action goes here */ 906 907 A similar trick will work for matching a 'foo' or a 908 'bar'-at-the-beginning-of-a-line. 909 910 911File: flex.info, Node: Matching, Next: Actions, Prev: Patterns, Up: Top 912 9137 How the Input Is Matched 914************************** 915 916When the generated scanner is run, it analyzes its input looking for 917strings which match any of its patterns. If it finds more than one 918match, it takes the one matching the most text (for trailing context 919rules, this includes the length of the trailing part, even though it 920will then be returned to the input). If it finds two or more matches of 921the same length, the rule listed first in the 'flex' input file is 922chosen. 923 924 Once the match is determined, the text corresponding to the match 925(called the "token") is made available in the global character pointer 926'yytext', and its length in the global integer 'yyleng'. The "action" 927corresponding to the matched pattern is then executed (*note Actions::), 928and then the remaining input is scanned for another match. 929 930 If no match is found, then the "default rule" is executed: the next 931character in the input is considered matched and copied to the standard 932output. Thus, the simplest valid 'flex' input is: 933 934 %% 935 936 which generates a scanner that simply copies its input (one character 937at a time) to its output. 938 939 Note that 'yytext' can be defined in two different ways: either as a 940character _pointer_ or as a character _array_. You can control which 941definition 'flex' uses by including one of the special directives 942'%pointer' or '%array' in the first (definitions) section of your flex 943input. The default is '%pointer', unless you use the '-l' lex 944compatibility option, in which case 'yytext' will be an array. The 945advantage of using '%pointer' is substantially faster scanning and no 946buffer overflow when matching very large tokens (unless you run out of 947dynamic memory). The disadvantage is that you are restricted in how 948your actions can modify 'yytext' (*note Actions::), and calls to the 949'unput()' function destroys the present contents of 'yytext', which can 950be a considerable porting headache when moving between different 'lex' 951versions. 952 953 The advantage of '%array' is that you can then modify 'yytext' to 954your heart's content, and calls to 'unput()' do not destroy 'yytext' 955(*note Actions::). Furthermore, existing 'lex' programs sometimes 956access 'yytext' externally using declarations of the form: 957 958 extern char yytext[]; 959 960 This definition is erroneous when used with '%pointer', but correct 961for '%array'. 962 963 The '%array' declaration defines 'yytext' to be an array of 'YYLMAX' 964characters, which defaults to a fairly large value. You can change the 965size by simply #define'ing 'YYLMAX' to a different value in the first 966section of your 'flex' input. As mentioned above, with '%pointer' 967yytext grows dynamically to accommodate large tokens. While this means 968your '%pointer' scanner can accommodate very large tokens (such as 969matching entire blocks of comments), bear in mind that each time the 970scanner must resize 'yytext' it also must rescan the entire token from 971the beginning, so matching such tokens can prove slow. 'yytext' 972presently does _not_ dynamically grow if a call to 'unput()' results in 973too much text being pushed back; instead, a run-time error results. 974 975 Also note that you cannot use '%array' with C++ scanner classes 976(*note Cxx::). 977 978 979File: flex.info, Node: Actions, Next: Generated Scanner, Prev: Matching, Up: Top 980 9818 Actions 982********* 983 984Each pattern in a rule has a corresponding "action", which can be any 985arbitrary C statement. The pattern ends at the first non-escaped 986whitespace character; the remainder of the line is its action. If the 987action is empty, then when the pattern is matched the input token is 988simply discarded. For example, here is the specification for a program 989which deletes all occurrences of 'zap me' from its input: 990 991 %% 992 "zap me" 993 994 This example will copy all other characters in the input to the 995output since they will be matched by the default rule. 996 997 Here is a program which compresses multiple blanks and tabs down to a 998single blank, and throws away whitespace found at the end of a line: 999 1000 %% 1001 [ \t]+ putchar( ' ' ); 1002 [ \t]+$ /* ignore this token */ 1003 1004 If the action contains a '{', then the action spans till the 1005balancing '}' is found, and the action may cross multiple lines. 'flex' 1006knows about C strings and comments and won't be fooled by braces found 1007within them, but also allows actions to begin with '%{' and will 1008consider the action to be all the text up to the next '%}' (regardless 1009of ordinary braces inside the action). 1010 1011 An action consisting solely of a vertical bar ('|') means "same as 1012the action for the next rule". See below for an illustration. 1013 1014 Actions can include arbitrary C code, including 'return' statements 1015to return a value to whatever routine called 'yylex()'. Each time 1016'yylex()' is called it continues processing tokens from where it last 1017left off until it either reaches the end of the file or executes a 1018return. 1019 1020 Actions are free to modify 'yytext' except for lengthening it (adding 1021characters to its end-these will overwrite later characters in the input 1022stream). This however does not apply when using '%array' (*note 1023Matching::). In that case, 'yytext' may be freely modified in any way. 1024 1025 Actions are free to modify 'yyleng' except they should not do so if 1026the action also includes use of 'yymore()' (see below). 1027 1028 There are a number of special directives which can be included within 1029an action: 1030 1031'ECHO' 1032 copies yytext to the scanner's output. 1033 1034'BEGIN' 1035 followed by the name of a start condition places the scanner in the 1036 corresponding start condition (see below). 1037 1038'REJECT' 1039 directs the scanner to proceed on to the "second best" rule which 1040 matched the input (or a prefix of the input). The rule is chosen 1041 as described above in *note Matching::, and 'yytext' and 'yyleng' 1042 set up appropriately. It may either be one which matched as much 1043 text as the originally chosen rule but came later in the 'flex' 1044 input file, or one which matched less text. For example, the 1045 following will both count the words in the input and call the 1046 routine 'special()' whenever 'frob' is seen: 1047 1048 int word_count = 0; 1049 %% 1050 1051 frob special(); REJECT; 1052 [^ \t\n]+ ++word_count; 1053 1054 Without the 'REJECT', any occurrences of 'frob' in the input would 1055 not be counted as words, since the scanner normally executes only 1056 one action per token. Multiple uses of 'REJECT' are allowed, each 1057 one finding the next best choice to the currently active rule. For 1058 example, when the following scanner scans the token 'abcd', it will 1059 write 'abcdabcaba' to the output: 1060 1061 %% 1062 a | 1063 ab | 1064 abc | 1065 abcd ECHO; REJECT; 1066 .|\n /* eat up any unmatched character */ 1067 1068 The first three rules share the fourth's action since they use the 1069 special '|' action. 1070 1071 'REJECT' is a particularly expensive feature in terms of scanner 1072 performance; if it is used in _any_ of the scanner's actions it 1073 will slow down _all_ of the scanner's matching. Furthermore, 1074 'REJECT' cannot be used with the '-Cf' or '-CF' options (*note 1075 Scanner Options::). 1076 1077 Note also that unlike the other special actions, 'REJECT' is a 1078 _branch_. Code immediately following it in the action will _not_ 1079 be executed. 1080 1081'yymore()' 1082 tells the scanner that the next time it matches a rule, the 1083 corresponding token should be _appended_ onto the current value of 1084 'yytext' rather than replacing it. For example, given the input 1085 'mega-kludge' the following will write 'mega-mega-kludge' to the 1086 output: 1087 1088 %% 1089 mega- ECHO; yymore(); 1090 kludge ECHO; 1091 1092 First 'mega-' is matched and echoed to the output. Then 'kludge' 1093 is matched, but the previous 'mega-' is still hanging around at the 1094 beginning of 'yytext' so the 'ECHO' for the 'kludge' rule will 1095 actually write 'mega-kludge'. 1096 1097 Two notes regarding use of 'yymore()'. First, 'yymore()' depends on 1098the value of 'yyleng' correctly reflecting the size of the current 1099token, so you must not modify 'yyleng' if you are using 'yymore()'. 1100Second, the presence of 'yymore()' in the scanner's action entails a 1101minor performance penalty in the scanner's matching speed. 1102 1103 'yyless(n)' returns all but the first 'n' characters of the current 1104token back to the input stream, where they will be rescanned when the 1105scanner looks for the next match. 'yytext' and 'yyleng' are adjusted 1106appropriately (e.g., 'yyleng' will now be equal to 'n'). For example, 1107on the input 'foobar' the following will write out 'foobarbar': 1108 1109 %% 1110 foobar ECHO; yyless(3); 1111 [a-z]+ ECHO; 1112 1113 An argument of 0 to 'yyless()' will cause the entire current input 1114string to be scanned again. Unless you've changed how the scanner will 1115subsequently process its input (using 'BEGIN', for example), this will 1116result in an endless loop. 1117 1118 Note that 'yyless()' is a macro and can only be used in the flex 1119input file, not from other source files. 1120 1121 'unput(c)' puts the character 'c' back onto the input stream. It 1122will be the next character scanned. The following action will take the 1123current token and cause it to be rescanned enclosed in parentheses. 1124 1125 { 1126 int i; 1127 /* Copy yytext because unput() trashes yytext */ 1128 char *yycopy = strdup( yytext ); 1129 unput( ')' ); 1130 for ( i = yyleng - 1; i >= 0; --i ) 1131 unput( yycopy[i] ); 1132 unput( '(' ); 1133 free( yycopy ); 1134 } 1135 1136 Note that since each 'unput()' puts the given character back at the 1137_beginning_ of the input stream, pushing back strings must be done 1138back-to-front. 1139 1140 An important potential problem when using 'unput()' is that if you 1141are using '%pointer' (the default), a call to 'unput()' _destroys_ the 1142contents of 'yytext', starting with its rightmost character and 1143devouring one character to the left with each call. If you need the 1144value of 'yytext' preserved after a call to 'unput()' (as in the above 1145example), you must either first copy it elsewhere, or build your scanner 1146using '%array' instead (*note Matching::). 1147 1148 Finally, note that you cannot put back 'EOF' to attempt to mark the 1149input stream with an end-of-file. 1150 1151 'input()' reads the next character from the input stream. For 1152example, the following is one way to eat up C comments: 1153 1154 %% 1155 "/*" { 1156 int c; 1157 1158 for ( ; ; ) 1159 { 1160 while ( (c = input()) != '*' && 1161 c != EOF ) 1162 ; /* eat up text of comment */ 1163 1164 if ( c == '*' ) 1165 { 1166 while ( (c = input()) == '*' ) 1167 ; 1168 if ( c == '/' ) 1169 break; /* found the end */ 1170 } 1171 1172 if ( c == EOF ) 1173 { 1174 error( "EOF in comment" ); 1175 break; 1176 } 1177 } 1178 } 1179 1180 (Note that if the scanner is compiled using 'C++', then 'input()' is 1181instead referred to as yyinput(), in order to avoid a name clash with 1182the 'C++' stream by the name of 'input'.) 1183 1184 'YY_FLUSH_BUFFER;' flushes the scanner's internal buffer so that the 1185next time the scanner attempts to match a token, it will first refill 1186the buffer using 'YY_INPUT()' (*note Generated Scanner::). This action 1187is a special case of the more general 'yy_flush_buffer;' function, 1188described below (*note Multiple Input Buffers::) 1189 1190 'yyterminate()' can be used in lieu of a return statement in an 1191action. It terminates the scanner and returns a 0 to the scanner's 1192caller, indicating "all done". By default, 'yyterminate()' is also 1193called when an end-of-file is encountered. It is a macro and may be 1194redefined. 1195 1196 1197File: flex.info, Node: Generated Scanner, Next: Start Conditions, Prev: Actions, Up: Top 1198 11999 The Generated Scanner 1200*********************** 1201 1202The output of 'flex' is the file 'lex.yy.c', which contains the scanning 1203routine 'yylex()', a number of tables used by it for matching tokens, 1204and a number of auxiliary routines and macros. By default, 'yylex()' is 1205declared as follows: 1206 1207 int yylex() 1208 { 1209 ... various definitions and the actions in here ... 1210 } 1211 1212 (If your environment supports function prototypes, then it will be 1213'int yylex( void )'.) This definition may be changed by defining the 1214'YY_DECL' macro. For example, you could use: 1215 1216 #define YY_DECL float lexscan( a, b ) float a, b; 1217 1218 to give the scanning routine the name 'lexscan', returning a float, 1219and taking two floats as arguments. Note that if you give arguments to 1220the scanning routine using a K&R-style/non-prototyped function 1221declaration, you must terminate the definition with a semi-colon (;). 1222 1223 'flex' generates 'C99' function definitions by default. However flex 1224does have the ability to generate obsolete, er, 'traditional', function 1225definitions. This is to support bootstrapping gcc on old systems. 1226Unfortunately, traditional definitions prevent us from using any 1227standard data types smaller than int (such as short, char, or bool) as 1228function arguments. For this reason, future versions of 'flex' may 1229generate standard C99 code only, leaving K&R-style functions to the 1230historians. Currently, if you do *not* want 'C99' definitions, then you 1231must use '%option noansi-definitions'. 1232 1233 Whenever 'yylex()' is called, it scans tokens from the global input 1234file 'yyin' (which defaults to stdin). It continues until it either 1235reaches an end-of-file (at which point it returns the value 0) or one of 1236its actions executes a 'return' statement. 1237 1238 If the scanner reaches an end-of-file, subsequent calls are undefined 1239unless either 'yyin' is pointed at a new input file (in which case 1240scanning continues from that file), or 'yyrestart()' is called. 1241'yyrestart()' takes one argument, a 'FILE *' pointer (which can be NULL, 1242if you've set up 'YY_INPUT' to scan from a source other than 'yyin'), 1243and initializes 'yyin' for scanning from that file. Essentially there 1244is no difference between just assigning 'yyin' to a new input file or 1245using 'yyrestart()' to do so; the latter is available for compatibility 1246with previous versions of 'flex', and because it can be used to switch 1247input files in the middle of scanning. It can also be used to throw 1248away the current input buffer, by calling it with an argument of 'yyin'; 1249but it would be better to use 'YY_FLUSH_BUFFER' (*note Actions::). Note 1250that 'yyrestart()' does _not_ reset the start condition to 'INITIAL' 1251(*note Start Conditions::). 1252 1253 If 'yylex()' stops scanning due to executing a 'return' statement in 1254one of the actions, the scanner may then be called again and it will 1255resume scanning where it left off. 1256 1257 By default (and for purposes of efficiency), the scanner uses 1258block-reads rather than simple 'getc()' calls to read characters from 1259'yyin'. The nature of how it gets its input can be controlled by 1260defining the 'YY_INPUT' macro. The calling sequence for 'YY_INPUT()' is 1261'YY_INPUT(buf,result,max_size)'. Its action is to place up to 1262'max_size' characters in the character array 'buf' and return in the 1263integer variable 'result' either the number of characters read or the 1264constant 'YY_NULL' (0 on Unix systems) to indicate 'EOF'. The default 1265'YY_INPUT' reads from the global file-pointer 'yyin'. 1266 1267 Here is a sample definition of 'YY_INPUT' (in the definitions section 1268of the input file): 1269 1270 %{ 1271 #define YY_INPUT(buf,result,max_size) \ 1272 { \ 1273 int c = getchar(); \ 1274 result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \ 1275 } 1276 %} 1277 1278 This definition will change the input processing to occur one 1279character at a time. 1280 1281 When the scanner receives an end-of-file indication from YY_INPUT, it 1282then checks the 'yywrap()' function. If 'yywrap()' returns false 1283(zero), then it is assumed that the function has gone ahead and set up 1284'yyin' to point to another input file, and scanning continues. If it 1285returns true (non-zero), then the scanner terminates, returning 0 to its 1286caller. Note that in either case, the start condition remains 1287unchanged; it does _not_ revert to 'INITIAL'. 1288 1289 If you do not supply your own version of 'yywrap()', then you must 1290either use '%option noyywrap' (in which case the scanner behaves as 1291though 'yywrap()' returned 1), or you must link with '-lfl' to obtain 1292the default version of the routine, which always returns 1. 1293 1294 For scanning from in-memory buffers (e.g., scanning strings), see 1295*note Scanning Strings::. *Note Multiple Input Buffers::. 1296 1297 The scanner writes its 'ECHO' output to the 'yyout' global (default, 1298'stdout'), which may be redefined by the user simply by assigning it to 1299some other 'FILE' pointer. 1300 1301 1302File: flex.info, Node: Start Conditions, Next: Multiple Input Buffers, Prev: Generated Scanner, Up: Top 1303 130410 Start Conditions 1305******************* 1306 1307'flex' provides a mechanism for conditionally activating rules. Any 1308rule whose pattern is prefixed with '<sc>' will only be active when the 1309scanner is in the "start condition" named 'sc'. For example, 1310 1311 <STRING>[^"]* { /* eat up the string body ... */ 1312 ... 1313 } 1314 1315 will be active only when the scanner is in the 'STRING' start 1316condition, and 1317 1318 <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */ 1319 ... 1320 } 1321 1322 will be active only when the current start condition is either 1323'INITIAL', 'STRING', or 'QUOTE'. 1324 1325 Start conditions are declared in the definitions (first) section of 1326the input using unindented lines beginning with either '%s' or '%x' 1327followed by a list of names. The former declares "inclusive" start 1328conditions, the latter "exclusive" start conditions. A start condition 1329is activated using the 'BEGIN' action. Until the next 'BEGIN' action is 1330executed, rules with the given start condition will be active and rules 1331with other start conditions will be inactive. If the start condition is 1332inclusive, then rules with no start conditions at all will also be 1333active. If it is exclusive, then _only_ rules qualified with the start 1334condition will be active. A set of rules contingent on the same 1335exclusive start condition describe a scanner which is independent of any 1336of the other rules in the 'flex' input. Because of this, exclusive 1337start conditions make it easy to specify "mini-scanners" which scan 1338portions of the input that are syntactically different from the rest 1339(e.g., comments). 1340 1341 If the distinction between inclusive and exclusive start conditions 1342is still a little vague, here's a simple example illustrating the 1343connection between the two. The set of rules: 1344 1345 %s example 1346 %% 1347 1348 <example>foo do_something(); 1349 1350 bar something_else(); 1351 1352 is equivalent to 1353 1354 %x example 1355 %% 1356 1357 <example>foo do_something(); 1358 1359 <INITIAL,example>bar something_else(); 1360 1361 Without the '<INITIAL,example>' qualifier, the 'bar' pattern in the 1362second example wouldn't be active (i.e., couldn't match) when in start 1363condition 'example'. If we just used '<example>' to qualify 'bar', 1364though, then it would only be active in 'example' and not in 'INITIAL', 1365while in the first example it's active in both, because in the first 1366example the 'example' start condition is an inclusive '(%s)' start 1367condition. 1368 1369 Also note that the special start-condition specifier '<*>' matches 1370every start condition. Thus, the above example could also have been 1371written: 1372 1373 %x example 1374 %% 1375 1376 <example>foo do_something(); 1377 1378 <*>bar something_else(); 1379 1380 The default rule (to 'ECHO' any unmatched character) remains active 1381in start conditions. It is equivalent to: 1382 1383 <*>.|\n ECHO; 1384 1385 'BEGIN(0)' returns to the original state where only the rules with no 1386start conditions are active. This state can also be referred to as the 1387start-condition 'INITIAL', so 'BEGIN(INITIAL)' is equivalent to 1388'BEGIN(0)'. (The parentheses around the start condition name are not 1389required but are considered good style.) 1390 1391 'BEGIN' actions can also be given as indented code at the beginning 1392of the rules section. For example, the following will cause the scanner 1393to enter the 'SPECIAL' start condition whenever 'yylex()' is called and 1394the global variable 'enter_special' is true: 1395 1396 int enter_special; 1397 1398 %x SPECIAL 1399 %% 1400 if ( enter_special ) 1401 BEGIN(SPECIAL); 1402 1403 <SPECIAL>blahblahblah 1404 ...more rules follow... 1405 1406 To illustrate the uses of start conditions, here is a scanner which 1407provides two different interpretations of a string like '123.456'. By 1408default it will treat it as three tokens, the integer '123', a dot 1409('.'), and the integer '456'. But if the string is preceded earlier in 1410the line by the string 'expect-floats' it will treat it as a single 1411token, the floating-point number '123.456': 1412 1413 %{ 1414 #include <math.h> 1415 %} 1416 %s expect 1417 1418 %% 1419 expect-floats BEGIN(expect); 1420 1421 <expect>[0-9]+.[0-9]+ { 1422 printf( "found a float, = %f\n", 1423 atof( yytext ) ); 1424 } 1425 <expect>\n { 1426 /* that's the end of the line, so 1427 * we need another "expect-number" 1428 * before we'll recognize any more 1429 * numbers 1430 */ 1431 BEGIN(INITIAL); 1432 } 1433 1434 [0-9]+ { 1435 printf( "found an integer, = %d\n", 1436 atoi( yytext ) ); 1437 } 1438 1439 "." printf( "found a dot\n" ); 1440 1441 Here is a scanner which recognizes (and discards) C comments while 1442maintaining a count of the current input line. 1443 1444 %x comment 1445 %% 1446 int line_num = 1; 1447 1448 "/*" BEGIN(comment); 1449 1450 <comment>[^*\n]* /* eat anything that's not a '*' */ 1451 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ 1452 <comment>\n ++line_num; 1453 <comment>"*"+"/" BEGIN(INITIAL); 1454 1455 This scanner goes to a bit of trouble to match as much text as 1456possible with each rule. In general, when attempting to write a 1457high-speed scanner try to match as much possible in each rule, as it's a 1458big win. 1459 1460 Note that start-conditions names are really integer values and can be 1461stored as such. Thus, the above could be extended in the following 1462fashion: 1463 1464 %x comment foo 1465 %% 1466 int line_num = 1; 1467 int comment_caller; 1468 1469 "/*" { 1470 comment_caller = INITIAL; 1471 BEGIN(comment); 1472 } 1473 1474 ... 1475 1476 <foo>"/*" { 1477 comment_caller = foo; 1478 BEGIN(comment); 1479 } 1480 1481 <comment>[^*\n]* /* eat anything that's not a '*' */ 1482 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ 1483 <comment>\n ++line_num; 1484 <comment>"*"+"/" BEGIN(comment_caller); 1485 1486 Furthermore, you can access the current start condition using the 1487integer-valued 'YY_START' macro. For example, the above assignments to 1488'comment_caller' could instead be written 1489 1490 comment_caller = YY_START; 1491 1492 Flex provides 'YYSTATE' as an alias for 'YY_START' (since that is 1493what's used by AT&T 'lex'). 1494 1495 For historical reasons, start conditions do not have their own 1496name-space within the generated scanner. The start condition names are 1497unmodified in the generated scanner and generated header. *Note 1498option-header::. *Note option-prefix::. 1499 1500 Finally, here's an example of how to match C-style quoted strings 1501using exclusive start conditions, including expanded escape sequences 1502(but not including checking for a string that's too long): 1503 1504 %x str 1505 1506 %% 1507 char string_buf[MAX_STR_CONST]; 1508 char *string_buf_ptr; 1509 1510 1511 \" string_buf_ptr = string_buf; BEGIN(str); 1512 1513 <str>\" { /* saw closing quote - all done */ 1514 BEGIN(INITIAL); 1515 *string_buf_ptr = '\0'; 1516 /* return string constant token type and 1517 * value to parser 1518 */ 1519 } 1520 1521 <str>\n { 1522 /* error - unterminated string constant */ 1523 /* generate error message */ 1524 } 1525 1526 <str>\\[0-7]{1,3} { 1527 /* octal escape sequence */ 1528 int result; 1529 1530 (void) sscanf( yytext + 1, "%o", &result ); 1531 1532 if ( result > 0xff ) 1533 /* error, constant is out-of-bounds */ 1534 1535 *string_buf_ptr++ = result; 1536 } 1537 1538 <str>\\[0-9]+ { 1539 /* generate error - bad escape sequence; something 1540 * like '\48' or '\0777777' 1541 */ 1542 } 1543 1544 <str>\\n *string_buf_ptr++ = '\n'; 1545 <str>\\t *string_buf_ptr++ = '\t'; 1546 <str>\\r *string_buf_ptr++ = '\r'; 1547 <str>\\b *string_buf_ptr++ = '\b'; 1548 <str>\\f *string_buf_ptr++ = '\f'; 1549 1550 <str>\\(.|\n) *string_buf_ptr++ = yytext[1]; 1551 1552 <str>[^\\\n\"]+ { 1553 char *yptr = yytext; 1554 1555 while ( *yptr ) 1556 *string_buf_ptr++ = *yptr++; 1557 } 1558 1559 Often, such as in some of the examples above, you wind up writing a 1560whole bunch of rules all preceded by the same start condition(s). Flex 1561makes this a little easier and cleaner by introducing a notion of start 1562condition "scope". A start condition scope is begun with: 1563 1564 <SCs>{ 1565 1566 where '<SCs>' is a list of one or more start conditions. Inside the 1567start condition scope, every rule automatically has the prefix '<SCs>' 1568applied to it, until a '}' which matches the initial '{'. So, for 1569example, 1570 1571 <ESC>{ 1572 "\\n" return '\n'; 1573 "\\r" return '\r'; 1574 "\\f" return '\f'; 1575 "\\0" return '\0'; 1576 } 1577 1578 is equivalent to: 1579 1580 <ESC>"\\n" return '\n'; 1581 <ESC>"\\r" return '\r'; 1582 <ESC>"\\f" return '\f'; 1583 <ESC>"\\0" return '\0'; 1584 1585 Start condition scopes may be nested. 1586 1587 The following routines are available for manipulating stacks of start 1588conditions: 1589 1590 -- Function: void yy_push_state ( int 'new_state' ) 1591 pushes the current start condition onto the top of the start 1592 condition stack and switches to 'new_state' as though you had used 1593 'BEGIN new_state' (recall that start condition names are also 1594 integers). 1595 1596 -- Function: void yy_pop_state () 1597 pops the top of the stack and switches to it via 'BEGIN'. 1598 1599 -- Function: int yy_top_state () 1600 returns the top of the stack without altering the stack's contents. 1601 1602 The start condition stack grows dynamically and so has no built-in 1603size limitation. If memory is exhausted, program execution aborts. 1604 1605 To use start condition stacks, your scanner must include a '%option 1606stack' directive (*note Scanner Options::). 1607 1608 1609File: flex.info, Node: Multiple Input Buffers, Next: EOF, Prev: Start Conditions, Up: Top 1610 161111 Multiple Input Buffers 1612************************* 1613 1614Some scanners (such as those which support "include" files) require 1615reading from several input streams. As 'flex' scanners do a large 1616amount of buffering, one cannot control where the next input will be 1617read from by simply writing a 'YY_INPUT()' which is sensitive to the 1618scanning context. 'YY_INPUT()' is only called when the scanner reaches 1619the end of its buffer, which may be a long time after scanning a 1620statement such as an 'include' statement which requires switching the 1621input source. 1622 1623 To negotiate these sorts of problems, 'flex' provides a mechanism for 1624creating and switching between multiple input buffers. An input buffer 1625is created by using: 1626 1627 -- Function: YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size ) 1628 1629 which takes a 'FILE' pointer and a size and creates a buffer 1630associated with the given file and large enough to hold 'size' 1631characters (when in doubt, use 'YY_BUF_SIZE' for the size). It returns 1632a 'YY_BUFFER_STATE' handle, which may then be passed to other routines 1633(see below). The 'YY_BUFFER_STATE' type is a pointer to an opaque 1634'struct yy_buffer_state' structure, so you may safely initialize 1635'YY_BUFFER_STATE' variables to '((YY_BUFFER_STATE) 0)' if you wish, and 1636also refer to the opaque structure in order to correctly declare input 1637buffers in source files other than that of your scanner. Note that the 1638'FILE' pointer in the call to 'yy_create_buffer' is only used as the 1639value of 'yyin' seen by 'YY_INPUT'. If you redefine 'YY_INPUT()' so it 1640no longer uses 'yyin', then you can safely pass a NULL 'FILE' pointer to 1641'yy_create_buffer'. You select a particular buffer to scan from using: 1642 1643 -- Function: void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer ) 1644 1645 The above function switches the scanner's input buffer so subsequent 1646tokens will come from 'new_buffer'. Note that 'yy_switch_to_buffer()' 1647may be used by 'yywrap()' to set things up for continued scanning, 1648instead of opening a new file and pointing 'yyin' at it. If you are 1649looking for a stack of input buffers, then you want to use 1650'yypush_buffer_state()' instead of this function. Note also that 1651switching input sources via either 'yy_switch_to_buffer()' or 'yywrap()' 1652does _not_ change the start condition. 1653 1654 -- Function: void yy_delete_buffer ( YY_BUFFER_STATE buffer ) 1655 1656 is used to reclaim the storage associated with a buffer. ('buffer' 1657can be NULL, in which case the routine does nothing.) You can also 1658clear the current contents of a buffer using: 1659 1660 -- Function: void yypush_buffer_state ( YY_BUFFER_STATE buffer ) 1661 1662 This function pushes the new buffer state onto an internal stack. 1663The pushed state becomes the new current state. The stack is maintained 1664by flex and will grow as required. This function is intended to be used 1665instead of 'yy_switch_to_buffer', when you want to change states, but 1666preserve the current state for later use. 1667 1668 -- Function: void yypop_buffer_state ( ) 1669 1670 This function removes the current state from the top of the stack, 1671and deletes it by calling 'yy_delete_buffer'. The next state on the 1672stack, if any, becomes the new current state. 1673 1674 -- Function: void yy_flush_buffer ( YY_BUFFER_STATE buffer ) 1675 1676 This function discards the buffer's contents, so the next time the 1677scanner attempts to match a token from the buffer, it will first fill 1678the buffer anew using 'YY_INPUT()'. 1679 1680 -- Function: YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size ) 1681 1682 is an alias for 'yy_create_buffer()', provided for compatibility with 1683the C++ use of 'new' and 'delete' for creating and destroying dynamic 1684objects. 1685 1686 'YY_CURRENT_BUFFER' macro returns a 'YY_BUFFER_STATE' handle to the 1687current buffer. It should not be used as an lvalue. 1688 1689 Here are two examples of using these features for writing a scanner 1690which expands include files (the '<<EOF>>' feature is discussed below). 1691 1692 This first example uses yypush_buffer_state and yypop_buffer_state. 1693Flex maintains the stack internally. 1694 1695 /* the "incl" state is used for picking up the name 1696 * of an include file 1697 */ 1698 %x incl 1699 %% 1700 include BEGIN(incl); 1701 1702 [a-z]+ ECHO; 1703 [^a-z\n]*\n? ECHO; 1704 1705 <incl>[ \t]* /* eat the whitespace */ 1706 <incl>[^ \t\n]+ { /* got the include file name */ 1707 yyin = fopen( yytext, "r" ); 1708 1709 if ( ! yyin ) 1710 error( ... ); 1711 1712 yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE )); 1713 1714 BEGIN(INITIAL); 1715 } 1716 1717 <<EOF>> { 1718 yypop_buffer_state(); 1719 1720 if ( !YY_CURRENT_BUFFER ) 1721 { 1722 yyterminate(); 1723 } 1724 } 1725 1726 The second example, below, does the same thing as the previous 1727example did, but manages its own input buffer stack manually (instead of 1728letting flex do it). 1729 1730 /* the "incl" state is used for picking up the name 1731 * of an include file 1732 */ 1733 %x incl 1734 1735 %{ 1736 #define MAX_INCLUDE_DEPTH 10 1737 YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH]; 1738 int include_stack_ptr = 0; 1739 %} 1740 1741 %% 1742 include BEGIN(incl); 1743 1744 [a-z]+ ECHO; 1745 [^a-z\n]*\n? ECHO; 1746 1747 <incl>[ \t]* /* eat the whitespace */ 1748 <incl>[^ \t\n]+ { /* got the include file name */ 1749 if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) 1750 { 1751 fprintf( stderr, "Includes nested too deeply" ); 1752 exit( 1 ); 1753 } 1754 1755 include_stack[include_stack_ptr++] = 1756 YY_CURRENT_BUFFER; 1757 1758 yyin = fopen( yytext, "r" ); 1759 1760 if ( ! yyin ) 1761 error( ... ); 1762 1763 yy_switch_to_buffer( 1764 yy_create_buffer( yyin, YY_BUF_SIZE ) ); 1765 1766 BEGIN(INITIAL); 1767 } 1768 1769 <<EOF>> { 1770 if ( --include_stack_ptr 0 ) 1771 { 1772 yyterminate(); 1773 } 1774 1775 else 1776 { 1777 yy_delete_buffer( YY_CURRENT_BUFFER ); 1778 yy_switch_to_buffer( 1779 include_stack[include_stack_ptr] ); 1780 } 1781 } 1782 1783 The following routines are available for setting up input buffers for 1784scanning in-memory strings instead of files. All of them create a new 1785input buffer for scanning the string, and return a corresponding 1786'YY_BUFFER_STATE' handle (which you should delete with 1787'yy_delete_buffer()' when done with it). They also switch to the new 1788buffer using 'yy_switch_to_buffer()', so the next call to 'yylex()' will 1789start scanning the string. 1790 1791 -- Function: YY_BUFFER_STATE yy_scan_string ( const char *str ) 1792 scans a NUL-terminated string. 1793 1794 -- Function: YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int len 1795 ) 1796 scans 'len' bytes (including possibly 'NUL's) starting at location 1797 'bytes'. 1798 1799 Note that both of these functions create and scan a _copy_ of the 1800string or bytes. (This may be desirable, since 'yylex()' modifies the 1801contents of the buffer it is scanning.) You can avoid the copy by 1802using: 1803 1804 -- Function: YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t 1805 size) 1806 which scans in place the buffer starting at 'base', consisting of 1807 'size' bytes, the last two bytes of which _must_ be 1808 'YY_END_OF_BUFFER_CHAR' (ASCII NUL). These last two bytes are not 1809 scanned; thus, scanning consists of 'base[0]' through 1810 'base[size-2]', inclusive. 1811 1812 If you fail to set up 'base' in this manner (i.e., forget the final 1813two 'YY_END_OF_BUFFER_CHAR' bytes), then 'yy_scan_buffer()' returns a 1814NULL pointer instead of creating a new input buffer. 1815 1816 -- Data type: yy_size_t 1817 is an integral type to which you can cast an integer expression 1818 reflecting the size of the buffer. 1819 1820 1821File: flex.info, Node: EOF, Next: Misc Macros, Prev: Multiple Input Buffers, Up: Top 1822 182312 End-of-File Rules 1824******************** 1825 1826The special rule '<<EOF>>' indicates actions which are to be taken when 1827an end-of-file is encountered and 'yywrap()' returns non-zero (i.e., 1828indicates no further files to process). The action must finish by doing 1829one of the following things: 1830 1831 * assigning 'yyin' to a new input file (in previous versions of 1832 'flex', after doing the assignment you had to call the special 1833 action 'YY_NEW_FILE'. This is no longer necessary.) 1834 1835 * executing a 'return' statement; 1836 1837 * executing the special 'yyterminate()' action. 1838 1839 * or, switching to a new buffer using 'yy_switch_to_buffer()' as 1840 shown in the example above. 1841 1842 <<EOF>> rules may not be used with other patterns; they may only be 1843qualified with a list of start conditions. If an unqualified <<EOF>> 1844rule is given, it applies to _all_ start conditions which do not already 1845have <<EOF>> actions. To specify an <<EOF>> rule for only the initial 1846start condition, use: 1847 1848 <INITIAL><<EOF>> 1849 1850 These rules are useful for catching things like unclosed comments. 1851An example: 1852 1853 %x quote 1854 %% 1855 1856 ...other rules for dealing with quotes... 1857 1858 <quote><<EOF>> { 1859 error( "unterminated quote" ); 1860 yyterminate(); 1861 } 1862 <<EOF>> { 1863 if ( *++filelist ) 1864 yyin = fopen( *filelist, "r" ); 1865 else 1866 yyterminate(); 1867 } 1868 1869 1870File: flex.info, Node: Misc Macros, Next: User Values, Prev: EOF, Up: Top 1871 187213 Miscellaneous Macros 1873*********************** 1874 1875The macro 'YY_USER_ACTION' can be defined to provide an action which is 1876always executed prior to the matched rule's action. For example, it 1877could be #define'd to call a routine to convert yytext to lower-case. 1878When 'YY_USER_ACTION' is invoked, the variable 'yy_act' gives the number 1879of the matched rule (rules are numbered starting with 1). Suppose you 1880want to profile how often each of your rules is matched. The following 1881would do the trick: 1882 1883 #define YY_USER_ACTION ++ctr[yy_act] 1884 1885 where 'ctr' is an array to hold the counts for the different rules. 1886Note that the macro 'YY_NUM_RULES' gives the total number of rules 1887(including the default rule), even if you use '-s)', so a correct 1888declaration for 'ctr' is: 1889 1890 int ctr[YY_NUM_RULES]; 1891 1892 The macro 'YY_USER_INIT' may be defined to provide an action which is 1893always executed before the first scan (and before the scanner's internal 1894initializations are done). For example, it could be used to call a 1895routine to read in a data table or open a logging file. 1896 1897 The macro 'yy_set_interactive(is_interactive)' can be used to control 1898whether the current buffer is considered "interactive". An interactive 1899buffer is processed more slowly, but must be used when the scanner's 1900input source is indeed interactive to avoid problems due to waiting to 1901fill buffers (see the discussion of the '-I' flag in *note Scanner 1902Options::). A non-zero value in the macro invocation marks the buffer 1903as interactive, a zero value as non-interactive. Note that use of this 1904macro overrides '%option always-interactive' or '%option 1905never-interactive' (*note Scanner Options::). 'yy_set_interactive()' 1906must be invoked prior to beginning to scan the buffer that is (or is 1907not) to be considered interactive. 1908 1909 The macro 'yy_set_bol(at_bol)' can be used to control whether the 1910current buffer's scanning context for the next token match is done as 1911though at the beginning of a line. A non-zero macro argument makes 1912rules anchored with '^' active, while a zero argument makes '^' rules 1913inactive. 1914 1915 The macro 'YY_AT_BOL()' returns true if the next token scanned from 1916the current buffer will have '^' rules active, false otherwise. 1917 1918 In the generated scanner, the actions are all gathered in one large 1919switch statement and separated using 'YY_BREAK', which may be redefined. 1920By default, it is simply a 'break', to separate each rule's action from 1921the following rule's. Redefining 'YY_BREAK' allows, for example, C++ 1922users to #define YY_BREAK to do nothing (while being very careful that 1923every rule ends with a 'break' or a 'return'!) to avoid suffering from 1924unreachable statement warnings where because a rule's action ends with 1925'return', the 'YY_BREAK' is inaccessible. 1926 1927 1928File: flex.info, Node: User Values, Next: Yacc, Prev: Misc Macros, Up: Top 1929 193014 Values Available To the User 1931******************************* 1932 1933This chapter summarizes the various values available to the user in the 1934rule actions. 1935 1936'char *yytext' 1937 holds the text of the current token. It may be modified but not 1938 lengthened (you cannot append characters to the end). 1939 1940 If the special directive '%array' appears in the first section of 1941 the scanner description, then 'yytext' is instead declared 'char 1942 yytext[YYLMAX]', where 'YYLMAX' is a macro definition that you can 1943 redefine in the first section if you don't like the default value 1944 (generally 8KB). Using '%array' results in somewhat slower 1945 scanners, but the value of 'yytext' becomes immune to calls to 1946 'unput()', which potentially destroy its value when 'yytext' is a 1947 character pointer. The opposite of '%array' is '%pointer', which 1948 is the default. 1949 1950 You cannot use '%array' when generating C++ scanner classes (the 1951 '-+' flag). 1952 1953'int yyleng' 1954 holds the length of the current token. 1955 1956'FILE *yyin' 1957 is the file which by default 'flex' reads from. It may be 1958 redefined but doing so only makes sense before scanning begins or 1959 after an EOF has been encountered. Changing it in the midst of 1960 scanning will have unexpected results since 'flex' buffers its 1961 input; use 'yyrestart()' instead. Once scanning terminates because 1962 an end-of-file has been seen, you can assign 'yyin' at the new 1963 input file and then call the scanner again to continue scanning. 1964 1965'void yyrestart( FILE *new_file )' 1966 may be called to point 'yyin' at the new input file. The 1967 switch-over to the new file is immediate (any previously 1968 buffered-up input is lost). Note that calling 'yyrestart()' with 1969 'yyin' as an argument thus throws away the current input buffer and 1970 continues scanning the same input file. 1971 1972'FILE *yyout' 1973 is the file to which 'ECHO' actions are done. It can be reassigned 1974 by the user. 1975 1976'YY_CURRENT_BUFFER' 1977 returns a 'YY_BUFFER_STATE' handle to the current buffer. 1978 1979'YY_START' 1980 returns an integer value corresponding to the current start 1981 condition. You can subsequently use this value with 'BEGIN' to 1982 return to that start condition. 1983 1984 1985File: flex.info, Node: Yacc, Next: Scanner Options, Prev: User Values, Up: Top 1986 198715 Interfacing with Yacc 1988************************ 1989 1990One of the main uses of 'flex' is as a companion to the 'yacc' 1991parser-generator. 'yacc' parsers expect to call a routine named 1992'yylex()' to find the next input token. The routine is supposed to 1993return the type of the next token as well as putting any associated 1994value in the global 'yylval'. To use 'flex' with 'yacc', one specifies 1995the '-d' option to 'yacc' to instruct it to generate the file 'y.tab.h' 1996containing definitions of all the '%tokens' appearing in the 'yacc' 1997input. This file is then included in the 'flex' scanner. For example, 1998if one of the tokens is 'TOK_NUMBER', part of the scanner might look 1999like: 2000 2001 %{ 2002 #include "y.tab.h" 2003 %} 2004 2005 %% 2006 2007 [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER; 2008 2009 2010File: flex.info, Node: Scanner Options, Next: Performance, Prev: Yacc, Up: Top 2011 201216 Scanner Options 2013****************** 2014 2015The various 'flex' options are categorized by function in the following 2016menu. If you want to lookup a particular option by name, *Note Index of 2017Scanner Options::. 2018 2019* Menu: 2020 2021* Options for Specifying Filenames:: 2022* Options Affecting Scanner Behavior:: 2023* Code-Level And API Options:: 2024* Options for Scanner Speed and Size:: 2025* Debugging Options:: 2026* Miscellaneous Options:: 2027 2028 Even though there are many scanner options, a typical scanner might 2029only specify the following options: 2030 2031 %option 8bit reentrant bison-bridge 2032 %option warn nodefault 2033 %option yylineno 2034 %option outfile="scanner.c" header-file="scanner.h" 2035 2036 The first line specifies the general type of scanner we want. The 2037second line specifies that we are being careful. The third line asks 2038flex to track line numbers. The last line tells flex what to name the 2039files. (The options can be specified in any order. We just divided 2040them.) 2041 2042 'flex' also provides a mechanism for controlling options within the 2043scanner specification itself, rather than from the flex command-line. 2044This is done by including '%option' directives in the first section of 2045the scanner specification. You can specify multiple options with a 2046single '%option' directive, and multiple directives in the first section 2047of your flex input file. 2048 2049 Most options are given simply as names, optionally preceded by the 2050word 'no' (with no intervening whitespace) to negate their meaning. The 2051names are the same as their long-option equivalents (but without the 2052leading '--' ). 2053 2054 'flex' scans your rule actions to determine whether you use the 2055'REJECT' or 'yymore()' features. The 'REJECT' and 'yymore' options are 2056available to override its decision as to whether you use the options, 2057either by setting them (e.g., '%option reject)' to indicate the feature 2058is indeed used, or unsetting them to indicate it actually is not used 2059(e.g., '%option noyymore)'. 2060 2061 A number of options are available for lint purists who want to 2062suppress the appearance of unneeded routines in the generated scanner. 2063Each of the following, if unset (e.g., '%option nounput'), results in 2064the corresponding routine not appearing in the generated scanner: 2065 2066 input, unput 2067 yy_push_state, yy_pop_state, yy_top_state 2068 yy_scan_buffer, yy_scan_bytes, yy_scan_string 2069 2070 yyget_extra, yyset_extra, yyget_leng, yyget_text, 2071 yyget_lineno, yyset_lineno, yyget_in, yyset_in, 2072 yyget_out, yyset_out, yyget_lval, yyset_lval, 2073 yyget_lloc, yyset_lloc, yyget_debug, yyset_debug 2074 2075 (though 'yy_push_state()' and friends won't appear anyway unless you 2076use '%option stack)'. 2077 2078 2079File: flex.info, Node: Options for Specifying Filenames, Next: Options Affecting Scanner Behavior, Prev: Scanner Options, Up: Scanner Options 2080 208116.1 Options for Specifying Filenames 2082===================================== 2083 2084'--header-file=FILE, '%option header-file="FILE"'' 2085 instructs flex to write a C header to 'FILE'. This file contains 2086 function prototypes, extern variables, and types used by the 2087 scanner. Only the external API is exported by the header file. 2088 Many macros that are usable from within scanner actions are not 2089 exported to the header file. This is due to namespace problems and 2090 the goal of a clean external API. 2091 2092 While in the header, the macro 'yyIN_HEADER' is defined, where 'yy' 2093 is substituted with the appropriate prefix. 2094 2095 The '--header-file' option is not compatible with the '--c++' 2096 option, since the C++ scanner provides its own header in 2097 'yyFlexLexer.h'. 2098 2099'-oFILE, --outfile=FILE, '%option outfile="FILE"'' 2100 directs flex to write the scanner to the file 'FILE' instead of 2101 'lex.yy.c'. If you combine '--outfile' with the '--stdout' option, 2102 then the scanner is written to 'stdout' but its '#line' directives 2103 (see the '-l' option above) refer to the file 'FILE'. 2104 2105'-t, --stdout, '%option stdout'' 2106 instructs 'flex' to write the scanner it generates to standard 2107 output instead of 'lex.yy.c'. 2108 2109'-SFILE, --skel=FILE' 2110 overrides the default skeleton file from which 'flex' constructs 2111 its scanners. You'll never need this option unless you are doing 2112 'flex' maintenance or development. 2113 2114'--tables-file=FILE' 2115 Write serialized scanner dfa tables to FILE. The generated scanner 2116 will not contain the tables, and requires them to be loaded at 2117 runtime. *Note serialization::. 2118 2119'--tables-verify' 2120 This option is for flex development. We document it here in case 2121 you stumble upon it by accident or in case you suspect some 2122 inconsistency in the serialized tables. Flex will serialize the 2123 scanner dfa tables but will also generate the in-code tables as it 2124 normally does. At runtime, the scanner will verify that the 2125 serialized tables match the in-code tables, instead of loading 2126 them. 2127 2128 2129File: flex.info, Node: Options Affecting Scanner Behavior, Next: Code-Level And API Options, Prev: Options for Specifying Filenames, Up: Scanner Options 2130 213116.2 Options Affecting Scanner Behavior 2132======================================= 2133 2134'-i, --case-insensitive, '%option case-insensitive'' 2135 instructs 'flex' to generate a "case-insensitive" scanner. The 2136 case of letters given in the 'flex' input patterns will be ignored, 2137 and tokens in the input will be matched regardless of case. The 2138 matched text given in 'yytext' will have the preserved case (i.e., 2139 it will not be folded). For tricky behavior, see *note case and 2140 character ranges::. 2141 2142'-l, --lex-compat, '%option lex-compat'' 2143 turns on maximum compatibility with the original AT&T 'lex' 2144 implementation. Note that this does not mean _full_ compatibility. 2145 Use of this option costs a considerable amount of performance, and 2146 it cannot be used with the '--c++', '--full', '--fast', '-Cf', or 2147 '-CF' options. For details on the compatibilities it provides, see 2148 *note Lex and Posix::. This option also results in the name 2149 'YY_FLEX_LEX_COMPAT' being '#define''d in the generated scanner. 2150 2151'-B, --batch, '%option batch'' 2152 instructs 'flex' to generate a "batch" scanner, the opposite of 2153 _interactive_ scanners generated by '--interactive' (see below). 2154 In general, you use '-B' when you are _certain_ that your scanner 2155 will never be used interactively, and you want to squeeze a 2156 _little_ more performance out of it. If your goal is instead to 2157 squeeze out a _lot_ more performance, you should be using the '-Cf' 2158 or '-CF' options, which turn on '--batch' automatically anyway. 2159 2160'-I, --interactive, '%option interactive'' 2161 instructs 'flex' to generate an interactive scanner. An 2162 interactive scanner is one that only looks ahead to decide what 2163 token has been matched if it absolutely must. It turns out that 2164 always looking one extra character ahead, even if the scanner has 2165 already seen enough text to disambiguate the current token, is a 2166 bit faster than only looking ahead when necessary. But scanners 2167 that always look ahead give dreadful interactive performance; for 2168 example, when a user types a newline, it is not recognized as a 2169 newline token until they enter _another_ token, which often means 2170 typing in another whole line. 2171 2172 'flex' scanners default to 'interactive' unless you use the '-Cf' 2173 or '-CF' table-compression options (*note Performance::). That's 2174 because if you're looking for high-performance you should be using 2175 one of these options, so if you didn't, 'flex' assumes you'd rather 2176 trade off a bit of run-time performance for intuitive interactive 2177 behavior. Note also that you _cannot_ use '--interactive' in 2178 conjunction with '-Cf' or '-CF'. Thus, this option is not really 2179 needed; it is on by default for all those cases in which it is 2180 allowed. 2181 2182 You can force a scanner to _not_ be interactive by using '--batch' 2183 2184'-7, --7bit, '%option 7bit'' 2185 instructs 'flex' to generate a 7-bit scanner, i.e., one which can 2186 only recognize 7-bit characters in its input. The advantage of 2187 using '--7bit' is that the scanner's tables can be up to half the 2188 size of those generated using the '--8bit'. The disadvantage is 2189 that such scanners often hang or crash if their input contains an 2190 8-bit character. 2191 2192 Note, however, that unless you generate your scanner using the 2193 '-Cf' or '-CF' table compression options, use of '--7bit' will save 2194 only a small amount of table space, and make your scanner 2195 considerably less portable. 'Flex''s default behavior is to 2196 generate an 8-bit scanner unless you use the '-Cf' or '-CF', in 2197 which case 'flex' defaults to generating 7-bit scanners unless your 2198 site was always configured to generate 8-bit scanners (as will 2199 often be the case with non-USA sites). You can tell whether flex 2200 generated a 7-bit or an 8-bit scanner by inspecting the flag 2201 summary in the '--verbose' output as described above. 2202 2203 Note that if you use '-Cfe' or '-CFe' 'flex' still defaults to 2204 generating an 8-bit scanner, since usually with these compression 2205 options full 8-bit tables are not much more expensive than 7-bit 2206 tables. 2207 2208'-8, --8bit, '%option 8bit'' 2209 instructs 'flex' to generate an 8-bit scanner, i.e., one which can 2210 recognize 8-bit characters. This flag is only needed for scanners 2211 generated using '-Cf' or '-CF', as otherwise flex defaults to 2212 generating an 8-bit scanner anyway. 2213 2214 See the discussion of '--7bit' above for 'flex''s default behavior 2215 and the tradeoffs between 7-bit and 8-bit scanners. 2216 2217'--default, '%option default'' 2218 generate the default rule. 2219 2220'--always-interactive, '%option always-interactive'' 2221 instructs flex to generate a scanner which always considers its 2222 input _interactive_. Normally, on each new input file the scanner 2223 calls 'isatty()' in an attempt to determine whether the scanner's 2224 input source is interactive and thus should be read a character at 2225 a time. When this option is used, however, then no such call is 2226 made. 2227 2228'--never-interactive, '--never-interactive'' 2229 instructs flex to generate a scanner which never considers its 2230 input interactive. This is the opposite of 'always-interactive'. 2231 2232'-X, --posix, '%option posix'' 2233 turns on maximum compatibility with the POSIX 1003.2-1992 2234 definition of 'lex'. Since 'flex' was originally designed to 2235 implement the POSIX definition of 'lex' this generally involves 2236 very few changes in behavior. At the current writing the known 2237 differences between 'flex' and the POSIX standard are: 2238 2239 * In POSIX and AT&T 'lex', the repeat operator, '{}', has lower 2240 precedence than concatenation (thus 'ab{3}' yields 'ababab'). 2241 Most POSIX utilities use an Extended Regular Expression (ERE) 2242 precedence that has the precedence of the repeat operator 2243 higher than concatenation (which causes 'ab{3}' to yield 2244 'abbb'). By default, 'flex' places the precedence of the 2245 repeat operator higher than concatenation which matches the 2246 ERE processing of other POSIX utilities. When either 2247 '--posix' or '-l' are specified, 'flex' will use the 2248 traditional AT&T and POSIX-compliant precedence for the repeat 2249 operator where concatenation has higher precedence than the 2250 repeat operator. 2251 2252'--stack, '%option stack'' 2253 enables the use of start condition stacks (*note Start 2254 Conditions::). 2255 2256'--stdinit, '%option stdinit'' 2257 if set (i.e., %option stdinit) initializes 'yyin' and 'yyout' to 2258 'stdin' and 'stdout', instead of the default of 'NULL'. Some 2259 existing 'lex' programs depend on this behavior, even though it is 2260 not compliant with ANSI C, which does not require 'stdin' and 2261 'stdout' to be compile-time constant. In a reentrant scanner, 2262 however, this is not a problem since initialization is performed in 2263 'yylex_init' at runtime. 2264 2265'--yylineno, '%option yylineno'' 2266 directs 'flex' to generate a scanner that maintains the number of 2267 the current line read from its input in the global variable 2268 'yylineno'. This option is implied by '%option lex-compat'. In a 2269 reentrant C scanner, the macro 'yylineno' is accessible regardless 2270 of the value of '%option yylineno', however, its value is not 2271 modified by 'flex' unless '%option yylineno' is enabled. 2272 2273'--yywrap, '%option yywrap'' 2274 if unset (i.e., '--noyywrap)', makes the scanner not call 2275 'yywrap()' upon an end-of-file, but simply assume that there are no 2276 more files to scan (until the user points 'yyin' at a new file and 2277 calls 'yylex()' again). 2278 2279 2280File: flex.info, Node: Code-Level And API Options, Next: Options for Scanner Speed and Size, Prev: Options Affecting Scanner Behavior, Up: Scanner Options 2281 228216.3 Code-Level And API Options 2283=============================== 2284 2285'--ansi-definitions, '%option ansi-definitions'' 2286 instruct flex to generate ANSI C99 definitions for functions. This 2287 option is enabled by default. If '%option noansi-definitions' is 2288 specified, then the obsolete style is generated. 2289 2290'--ansi-prototypes, '%option ansi-prototypes'' 2291 instructs flex to generate ANSI C99 prototypes for functions. This 2292 option is enabled by default. If 'noansi-prototypes' is specified, 2293 then prototypes will have empty parameter lists. 2294 2295'--bison-bridge, '%option bison-bridge'' 2296 instructs flex to generate a C scanner that is meant to be called 2297 by a 'GNU bison' parser. The scanner has minor API changes for 2298 'bison' compatibility. In particular, the declaration of 'yylex' 2299 is modified to take an additional parameter, 'yylval'. *Note Bison 2300 Bridge::. 2301 2302'--bison-locations, '%option bison-locations'' 2303 instruct flex that 'GNU bison' '%locations' are being used. This 2304 means 'yylex' will be passed an additional parameter, 'yylloc'. 2305 This option implies '%option bison-bridge'. *Note Bison Bridge::. 2306 2307'-L, --noline, '%option noline'' 2308 instructs 'flex' not to generate '#line' directives. Without this 2309 option, 'flex' peppers the generated scanner with '#line' 2310 directives so error messages in the actions will be correctly 2311 located with respect to either the original 'flex' input file (if 2312 the errors are due to code in the input file), or 'lex.yy.c' (if 2313 the errors are 'flex''s fault - you should report these sorts of 2314 errors to the email address given in *note Reporting Bugs::). 2315 2316'-R, --reentrant, '%option reentrant'' 2317 instructs flex to generate a reentrant C scanner. The generated 2318 scanner may safely be used in a multi-threaded environment. The 2319 API for a reentrant scanner is different than for a non-reentrant 2320 scanner *note Reentrant::). Because of the API difference between 2321 reentrant and non-reentrant 'flex' scanners, non-reentrant flex 2322 code must be modified before it is suitable for use with this 2323 option. This option is not compatible with the '--c++' option. 2324 2325 The option '--reentrant' does not affect the performance of the 2326 scanner. 2327 2328'-+, --c++, '%option c++'' 2329 specifies that you want flex to generate a C++ scanner class. 2330 *Note Cxx::, for details. 2331 2332'--array, '%option array'' 2333 specifies that you want yytext to be an array instead of a char* 2334 2335'--pointer, '%option pointer'' 2336 specify that 'yytext' should be a 'char *', not an array. This 2337 default is 'char *'. 2338 2339'-PPREFIX, --prefix=PREFIX, '%option prefix="PREFIX"'' 2340 changes the default 'yy' prefix used by 'flex' for all 2341 globally-visible variable and function names to instead be 2342 'PREFIX'. For example, '--prefix=foo' changes the name of 'yytext' 2343 to 'footext'. It also changes the name of the default output file 2344 from 'lex.yy.c' to 'lex.foo.c'. Here is a partial list of the 2345 names affected: 2346 2347 yy_create_buffer 2348 yy_delete_buffer 2349 yy_flex_debug 2350 yy_init_buffer 2351 yy_flush_buffer 2352 yy_load_buffer_state 2353 yy_switch_to_buffer 2354 yyin 2355 yyleng 2356 yylex 2357 yylineno 2358 yyout 2359 yyrestart 2360 yytext 2361 yywrap 2362 yyalloc 2363 yyrealloc 2364 yyfree 2365 2366 (If you are using a C++ scanner, then only 'yywrap' and 2367 'yyFlexLexer' are affected.) Within your scanner itself, you can 2368 still refer to the global variables and functions using either 2369 version of their name; but externally, they have the modified name. 2370 2371 This option lets you easily link together multiple 'flex' programs 2372 into the same executable. Note, though, that using this option 2373 also renames 'yywrap()', so you now _must_ either provide your own 2374 (appropriately-named) version of the routine for your scanner, or 2375 use '%option noyywrap', as linking with '-lfl' no longer provides 2376 one for you by default. 2377 2378'--main, '%option main'' 2379 directs flex to provide a default 'main()' program for the scanner, 2380 which simply calls 'yylex()'. This option implies 'noyywrap' (see 2381 below). 2382 2383'--nounistd, '%option nounistd'' 2384 suppresses inclusion of the non-ANSI header file 'unistd.h'. This 2385 option is meant to target environments in which 'unistd.h' does not 2386 exist. Be aware that certain options may cause flex to generate 2387 code that relies on functions normally found in 'unistd.h', (e.g. 2388 'isatty()', 'read()'.) If you wish to use these functions, you 2389 will have to inform your compiler where to find them. *Note 2390 option-always-interactive::. *Note option-read::. 2391 2392'--yyclass=NAME, '%option yyclass="NAME"'' 2393 only applies when generating a C++ scanner (the '--c++' option). 2394 It informs 'flex' that you have derived 'NAME' as a subclass of 2395 'yyFlexLexer', so 'flex' will place your actions in the member 2396 function 'foo::yylex()' instead of 'yyFlexLexer::yylex()'. It also 2397 generates a 'yyFlexLexer::yylex()' member function that emits a 2398 run-time error (by invoking 'yyFlexLexer::LexerError())' if called. 2399 *Note Cxx::. 2400 2401 2402File: flex.info, Node: Options for Scanner Speed and Size, Next: Debugging Options, Prev: Code-Level And API Options, Up: Scanner Options 2403 240416.4 Options for Scanner Speed and Size 2405======================================= 2406 2407'-C[aefFmr]' 2408 controls the degree of table compression and, more generally, 2409 trade-offs between small scanners and fast scanners. 2410 2411 '-C' 2412 A lone '-C' specifies that the scanner tables should be 2413 compressed but neither equivalence classes nor 2414 meta-equivalence classes should be used. 2415 2416 '-Ca, --align, '%option align'' 2417 ("align") instructs flex to trade off larger tables in the 2418 generated scanner for faster performance because the elements 2419 of the tables are better aligned for memory access and 2420 computation. On some RISC architectures, fetching and 2421 manipulating longwords is more efficient than with 2422 smaller-sized units such as shortwords. This option can 2423 quadruple the size of the tables used by your scanner. 2424 2425 '-Ce, --ecs, '%option ecs'' 2426 directs 'flex' to construct "equivalence classes", i.e., sets 2427 of characters which have identical lexical properties (for 2428 example, if the only appearance of digits in the 'flex' input 2429 is in the character class "[0-9]" then the digits '0', '1', 2430 ..., '9' will all be put in the same equivalence class). 2431 Equivalence classes usually give dramatic reductions in the 2432 final table/object file sizes (typically a factor of 2-5) and 2433 are pretty cheap performance-wise (one array look-up per 2434 character scanned). 2435 2436 '-Cf' 2437 specifies that the "full" scanner tables should be generated - 2438 'flex' should not compress the tables by taking advantages of 2439 similar transition functions for different states. 2440 2441 '-CF' 2442 specifies that the alternate fast scanner representation 2443 (described above under the '--fast' flag) should be used. 2444 This option cannot be used with '--c++'. 2445 2446 '-Cm, --meta-ecs, '%option meta-ecs'' 2447 directs 'flex' to construct "meta-equivalence classes", which 2448 are sets of equivalence classes (or characters, if equivalence 2449 classes are not being used) that are commonly used together. 2450 Meta-equivalence classes are often a big win when using 2451 compressed tables, but they have a moderate performance impact 2452 (one or two 'if' tests and one array look-up per character 2453 scanned). 2454 2455 '-Cr, --read, '%option read'' 2456 causes the generated scanner to _bypass_ use of the standard 2457 I/O library ('stdio') for input. Instead of calling 'fread()' 2458 or 'getc()', the scanner will use the 'read()' system call, 2459 resulting in a performance gain which varies from system to 2460 system, but in general is probably negligible unless you are 2461 also using '-Cf' or '-CF'. Using '-Cr' can cause strange 2462 behavior if, for example, you read from 'yyin' using 'stdio' 2463 prior to calling the scanner (because the scanner will miss 2464 whatever text your previous reads left in the 'stdio' input 2465 buffer). '-Cr' has no effect if you define 'YY_INPUT()' 2466 (*note Generated Scanner::). 2467 2468 The options '-Cf' or '-CF' and '-Cm' do not make sense together - 2469 there is no opportunity for meta-equivalence classes if the table 2470 is not being compressed. Otherwise the options may be freely 2471 mixed, and are cumulative. 2472 2473 The default setting is '-Cem', which specifies that 'flex' should 2474 generate equivalence classes and meta-equivalence classes. This 2475 setting provides the highest degree of table compression. You can 2476 trade off faster-executing scanners at the cost of larger tables 2477 with the following generally being true: 2478 2479 slowest & smallest 2480 -Cem 2481 -Cm 2482 -Ce 2483 -C 2484 -C{f,F}e 2485 -C{f,F} 2486 -C{f,F}a 2487 fastest & largest 2488 2489 Note that scanners with the smallest tables are usually generated 2490 and compiled the quickest, so during development you will usually 2491 want to use the default, maximal compression. 2492 2493 '-Cfe' is often a good compromise between speed and size for 2494 production scanners. 2495 2496'-f, --full, '%option full'' 2497 specifies "fast scanner". No table compression is done and 'stdio' 2498 is bypassed. The result is large but fast. This option is 2499 equivalent to '--Cfr' 2500 2501'-F, --fast, '%option fast'' 2502 specifies that the _fast_ scanner table representation should be 2503 used (and 'stdio' bypassed). This representation is about as fast 2504 as the full table representation '--full', and for some sets of 2505 patterns will be considerably smaller (and for others, larger). In 2506 general, if the pattern set contains both _keywords_ and a 2507 catch-all, _identifier_ rule, such as in the set: 2508 2509 "case" return TOK_CASE; 2510 "switch" return TOK_SWITCH; 2511 ... 2512 "default" return TOK_DEFAULT; 2513 [a-z]+ return TOK_ID; 2514 2515 then you're better off using the full table representation. If 2516 only the _identifier_ rule is present and you then use a hash table 2517 or some such to detect the keywords, you're better off using 2518 '--fast'. 2519 2520 This option is equivalent to '-CFr'. It cannot be used with 2521 '--c++'. 2522 2523 2524File: flex.info, Node: Debugging Options, Next: Miscellaneous Options, Prev: Options for Scanner Speed and Size, Up: Scanner Options 2525 252616.5 Debugging Options 2527====================== 2528 2529'-b, --backup, '%option backup'' 2530 Generate backing-up information to 'lex.backup'. This is a list of 2531 scanner states which require backing up and the input characters on 2532 which they do so. By adding rules one can remove backing-up 2533 states. If _all_ backing-up states are eliminated and '-Cf' or 2534 '-CF' is used, the generated scanner will run faster (see the 2535 '--perf-report' flag). Only users who wish to squeeze every last 2536 cycle out of their scanners need worry about this option. (*note 2537 Performance::). 2538 2539'-d, --debug, '%option debug'' 2540 makes the generated scanner run in "debug" mode. Whenever a 2541 pattern is recognized and the global variable 'yy_flex_debug' is 2542 non-zero (which is the default), the scanner will write to 'stderr' 2543 a line of the form: 2544 2545 -accepting rule at line 53 ("the matched text") 2546 2547 The line number refers to the location of the rule in the file 2548 defining the scanner (i.e., the file that was fed to flex). 2549 Messages are also generated when the scanner backs up, accepts the 2550 default rule, reaches the end of its input buffer (or encounters a 2551 NUL; at this point, the two look the same as far as the scanner's 2552 concerned), or reaches an end-of-file. 2553 2554'-p, --perf-report, '%option perf-report'' 2555 generates a performance report to 'stderr'. The report consists of 2556 comments regarding features of the 'flex' input file which will 2557 cause a serious loss of performance in the resulting scanner. If 2558 you give the flag twice, you will also get comments regarding 2559 features that lead to minor performance losses. 2560 2561 Note that the use of 'REJECT', and variable trailing context (*note 2562 Limitations::) entails a substantial performance penalty; use of 2563 'yymore()', the '^' operator, and the '--interactive' flag entail 2564 minor performance penalties. 2565 2566'-s, --nodefault, '%option nodefault'' 2567 causes the _default rule_ (that unmatched scanner input is echoed 2568 to 'stdout)' to be suppressed. If the scanner encounters input 2569 that does not match any of its rules, it aborts with an error. 2570 This option is useful for finding holes in a scanner's rule set. 2571 2572'-T, --trace, '%option trace'' 2573 makes 'flex' run in "trace" mode. It will generate a lot of 2574 messages to 'stderr' concerning the form of the input and the 2575 resultant non-deterministic and deterministic finite automata. 2576 This option is mostly for use in maintaining 'flex'. 2577 2578'-w, --nowarn, '%option nowarn'' 2579 suppresses warning messages. 2580 2581'-v, --verbose, '%option verbose'' 2582 specifies that 'flex' should write to 'stderr' a summary of 2583 statistics regarding the scanner it generates. Most of the 2584 statistics are meaningless to the casual 'flex' user, but the first 2585 line identifies the version of 'flex' (same as reported by 2586 '--version'), and the next line the flags used when generating the 2587 scanner, including those that are on by default. 2588 2589'--warn, '%option warn'' 2590 warn about certain things. In particular, if the default rule can 2591 be matched but no default rule has been given, the flex will warn 2592 you. We recommend using this option always. 2593 2594 2595File: flex.info, Node: Miscellaneous Options, Prev: Debugging Options, Up: Scanner Options 2596 259716.6 Miscellaneous Options 2598========================== 2599 2600'-c' 2601 A do-nothing option included for POSIX compliance. 2602 2603'-h, -?, --help' 2604 generates a "help" summary of 'flex''s options to 'stdout' and then 2605 exits. 2606 2607'-n' 2608 Another do-nothing option included for POSIX compliance. 2609 2610'-V, --version' 2611 prints the version number to 'stdout' and exits. 2612 2613 2614File: flex.info, Node: Performance, Next: Cxx, Prev: Scanner Options, Up: Top 2615 261617 Performance Considerations 2617***************************** 2618 2619The main design goal of 'flex' is that it generate high-performance 2620scanners. It has been optimized for dealing well with large sets of 2621rules. Aside from the effects on scanner speed of the table compression 2622'-C' options outlined above, there are a number of options/actions which 2623degrade performance. These are, from most expensive to least: 2624 2625 REJECT 2626 arbitrary trailing context 2627 2628 pattern sets that require backing up 2629 %option yylineno 2630 %array 2631 2632 %option interactive 2633 %option always-interactive 2634 2635 ^ beginning-of-line operator 2636 yymore() 2637 2638 with the first two all being quite expensive and the last two being 2639quite cheap. Note also that 'unput()' is implemented as a routine call 2640that potentially does quite a bit of work, while 'yyless()' is a 2641quite-cheap macro. So if you are just putting back some excess text you 2642scanned, use 'yyless()'. 2643 2644 'REJECT' should be avoided at all costs when performance is 2645important. It is a particularly expensive option. 2646 2647 There is one case when '%option yylineno' can be expensive. That is 2648when your patterns match long tokens that could _possibly_ contain a 2649newline character. There is no performance penalty for rules that can 2650not possibly match newlines, since flex does not need to check them for 2651newlines. In general, you should avoid rules such as '[^f]+', which 2652match very long tokens, including newlines, and may possibly match your 2653entire file! A better approach is to separate '[^f]+' into two rules: 2654 2655 %option yylineno 2656 %% 2657 [^f\n]+ 2658 \n+ 2659 2660 The above scanner does not incur a performance penalty. 2661 2662 Getting rid of backing up is messy and often may be an enormous 2663amount of work for a complicated scanner. In principal, one begins by 2664using the '-b' flag to generate a 'lex.backup' file. For example, on 2665the input: 2666 2667 %% 2668 foo return TOK_KEYWORD; 2669 foobar return TOK_KEYWORD; 2670 2671 the file looks like: 2672 2673 State #6 is non-accepting - 2674 associated rule line numbers: 2675 2 3 2676 out-transitions: [ o ] 2677 jam-transitions: EOF [ \001-n p-\177 ] 2678 2679 State #8 is non-accepting - 2680 associated rule line numbers: 2681 3 2682 out-transitions: [ a ] 2683 jam-transitions: EOF [ \001-` b-\177 ] 2684 2685 State #9 is non-accepting - 2686 associated rule line numbers: 2687 3 2688 out-transitions: [ r ] 2689 jam-transitions: EOF [ \001-q s-\177 ] 2690 2691 Compressed tables always back up. 2692 2693 The first few lines tell us that there's a scanner state in which it 2694can make a transition on an 'o' but not on any other character, and that 2695in that state the currently scanned text does not match any rule. The 2696state occurs when trying to match the rules found at lines 2 and 3 in 2697the input file. If the scanner is in that state and then reads 2698something other than an 'o', it will have to back up to find a rule 2699which is matched. With a bit of headscratching one can see that this 2700must be the state it's in when it has seen 'fo'. When this has 2701happened, if anything other than another 'o' is seen, the scanner will 2702have to back up to simply match the 'f' (by the default rule). 2703 2704 The comment regarding State #8 indicates there's a problem when 2705'foob' has been scanned. Indeed, on any character other than an 'a', 2706the scanner will have to back up to accept "foo". Similarly, the 2707comment for State #9 concerns when 'fooba' has been scanned and an 'r' 2708does not follow. 2709 2710 The final comment reminds us that there's no point going to all the 2711trouble of removing backing up from the rules unless we're using '-Cf' 2712or '-CF', since there's no performance gain doing so with compressed 2713scanners. 2714 2715 The way to remove the backing up is to add "error" rules: 2716 2717 %% 2718 foo return TOK_KEYWORD; 2719 foobar return TOK_KEYWORD; 2720 2721 fooba | 2722 foob | 2723 fo { 2724 /* false alarm, not really a keyword */ 2725 return TOK_ID; 2726 } 2727 2728 Eliminating backing up among a list of keywords can also be done 2729using a "catch-all" rule: 2730 2731 %% 2732 foo return TOK_KEYWORD; 2733 foobar return TOK_KEYWORD; 2734 2735 [a-z]+ return TOK_ID; 2736 2737 This is usually the best solution when appropriate. 2738 2739 Backing up messages tend to cascade. With a complicated set of rules 2740it's not uncommon to get hundreds of messages. If one can decipher 2741them, though, it often only takes a dozen or so rules to eliminate the 2742backing up (though it's easy to make a mistake and have an error rule 2743accidentally match a valid token. A possible future 'flex' feature will 2744be to automatically add rules to eliminate backing up). 2745 2746 It's important to keep in mind that you gain the benefits of 2747eliminating backing up only if you eliminate _every_ instance of backing 2748up. Leaving just one means you gain nothing. 2749 2750 _Variable_ trailing context (where both the leading and trailing 2751parts do not have a fixed length) entails almost the same performance 2752loss as 'REJECT' (i.e., substantial). So when possible a rule like: 2753 2754 %% 2755 mouse|rat/(cat|dog) run(); 2756 2757 is better written: 2758 2759 %% 2760 mouse/cat|dog run(); 2761 rat/cat|dog run(); 2762 2763 or as 2764 2765 %% 2766 mouse|rat/cat run(); 2767 mouse|rat/dog run(); 2768 2769 Note that here the special '|' action does _not_ provide any savings, 2770and can even make things worse (*note Limitations::). 2771 2772 Another area where the user can increase a scanner's performance (and 2773one that's easier to implement) arises from the fact that the longer the 2774tokens matched, the faster the scanner will run. This is because with 2775long tokens the processing of most input characters takes place in the 2776(short) inner scanning loop, and does not often have to go through the 2777additional work of setting up the scanning environment (e.g., 'yytext') 2778for the action. Recall the scanner for C comments: 2779 2780 %x comment 2781 %% 2782 int line_num = 1; 2783 2784 "/*" BEGIN(comment); 2785 2786 <comment>[^*\n]* 2787 <comment>"*"+[^*/\n]* 2788 <comment>\n ++line_num; 2789 <comment>"*"+"/" BEGIN(INITIAL); 2790 2791 This could be sped up by writing it as: 2792 2793 %x comment 2794 %% 2795 int line_num = 1; 2796 2797 "/*" BEGIN(comment); 2798 2799 <comment>[^*\n]* 2800 <comment>[^*\n]*\n ++line_num; 2801 <comment>"*"+[^*/\n]* 2802 <comment>"*"+[^*/\n]*\n ++line_num; 2803 <comment>"*"+"/" BEGIN(INITIAL); 2804 2805 Now instead of each newline requiring the processing of another 2806action, recognizing the newlines is distributed over the other rules to 2807keep the matched text as long as possible. Note that _adding_ rules 2808does _not_ slow down the scanner! The speed of the scanner is 2809independent of the number of rules or (modulo the considerations given 2810at the beginning of this section) how complicated the rules are with 2811regard to operators such as '*' and '|'. 2812 2813 A final example in speeding up a scanner: suppose you want to scan 2814through a file containing identifiers and keywords, one per line and 2815with no other extraneous characters, and recognize all the keywords. A 2816natural first approach is: 2817 2818 %% 2819 asm | 2820 auto | 2821 break | 2822 ... etc ... 2823 volatile | 2824 while /* it's a keyword */ 2825 2826 .|\n /* it's not a keyword */ 2827 2828 To eliminate the back-tracking, introduce a catch-all rule: 2829 2830 %% 2831 asm | 2832 auto | 2833 break | 2834 ... etc ... 2835 volatile | 2836 while /* it's a keyword */ 2837 2838 [a-z]+ | 2839 .|\n /* it's not a keyword */ 2840 2841 Now, if it's guaranteed that there's exactly one word per line, then 2842we can reduce the total number of matches by a half by merging in the 2843recognition of newlines with that of the other tokens: 2844 2845 %% 2846 asm\n | 2847 auto\n | 2848 break\n | 2849 ... etc ... 2850 volatile\n | 2851 while\n /* it's a keyword */ 2852 2853 [a-z]+\n | 2854 .|\n /* it's not a keyword */ 2855 2856 One has to be careful here, as we have now reintroduced backing up 2857into the scanner. In particular, while _we_ know that there will never 2858be any characters in the input stream other than letters or newlines, 2859'flex' can't figure this out, and it will plan for possibly needing to 2860back up when it has scanned a token like 'auto' and then the next 2861character is something other than a newline or a letter. Previously it 2862would then just match the 'auto' rule and be done, but now it has no 2863'auto' rule, only a 'auto\n' rule. To eliminate the possibility of 2864backing up, we could either duplicate all rules but without final 2865newlines, or, since we never expect to encounter such an input and 2866therefore don't how it's classified, we can introduce one more catch-all 2867rule, this one which doesn't include a newline: 2868 2869 %% 2870 asm\n | 2871 auto\n | 2872 break\n | 2873 ... etc ... 2874 volatile\n | 2875 while\n /* it's a keyword */ 2876 2877 [a-z]+\n | 2878 [a-z]+ | 2879 .|\n /* it's not a keyword */ 2880 2881 Compiled with '-Cf', this is about as fast as one can get a 'flex' 2882scanner to go for this particular problem. 2883 2884 A final note: 'flex' is slow when matching 'NUL's, particularly when 2885a token contains multiple 'NUL's. It's best to write rules which match 2886_short_ amounts of text if it's anticipated that the text will often 2887include 'NUL's. 2888 2889 Another final note regarding performance: as mentioned in *note 2890Matching::, dynamically resizing 'yytext' to accommodate huge tokens is 2891a slow process because it presently requires that the (huge) token be 2892rescanned from the beginning. Thus if performance is vital, you should 2893attempt to match "large" quantities of text but not "huge" quantities, 2894where the cutoff between the two is at about 8K characters per token. 2895 2896 2897File: flex.info, Node: Cxx, Next: Reentrant, Prev: Performance, Up: Top 2898 289918 Generating C++ Scanners 2900************************** 2901 2902*IMPORTANT*: the present form of the scanning class is _experimental_ 2903and may change considerably between major releases. 2904 2905 'flex' provides two different ways to generate scanners for use with 2906C++. The first way is to simply compile a scanner generated by 'flex' 2907using a C++ compiler instead of a C compiler. You should not encounter 2908any compilation errors (*note Reporting Bugs::). You can then use C++ 2909code in your rule actions instead of C code. Note that the default 2910input source for your scanner remains 'yyin', and default echoing is 2911still done to 'yyout'. Both of these remain 'FILE *' variables and not 2912C++ _streams_. 2913 2914 You can also use 'flex' to generate a C++ scanner class, using the 2915'-+' option (or, equivalently, '%option c++)', which is automatically 2916specified if the name of the 'flex' executable ends in a '+', such as 2917'flex++'. When using this option, 'flex' defaults to generating the 2918scanner to the file 'lex.yy.cc' instead of 'lex.yy.c'. The generated 2919scanner includes the header file 'FlexLexer.h', which defines the 2920interface to two C++ classes. 2921 2922 The first class in 'FlexLexer.h', 'FlexLexer', provides an abstract 2923base class defining the general scanner class interface. It provides 2924the following member functions: 2925 2926'const char* YYText()' 2927 returns the text of the most recently matched token, the equivalent 2928 of 'yytext'. 2929 2930'int YYLeng()' 2931 returns the length of the most recently matched token, the 2932 equivalent of 'yyleng'. 2933 2934'int lineno() const' 2935 returns the current input line number (see '%option yylineno)', or 2936 '1' if '%option yylineno' was not used. 2937 2938'void set_debug( int flag )' 2939 sets the debugging flag for the scanner, equivalent to assigning to 2940 'yy_flex_debug' (*note Scanner Options::). Note that you must 2941 build the scanner using '%option debug' to include debugging 2942 information in it. 2943 2944'int debug() const' 2945 returns the current setting of the debugging flag. 2946 2947 Also provided are member functions equivalent to 2948'yy_switch_to_buffer()', 'yy_create_buffer()' (though the first argument 2949is an 'istream&' object reference and not a 'FILE*)', 2950'yy_flush_buffer()', 'yy_delete_buffer()', and 'yyrestart()' (again, the 2951first argument is a 'istream&' object reference). 2952 2953 The second class defined in 'FlexLexer.h' is 'yyFlexLexer', which is 2954derived from 'FlexLexer'. It defines the following additional member 2955functions: 2956 2957'yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )' 2958'yyFlexLexer( istream& arg_yyin, ostream& arg_yyout )' 2959 constructs a 'yyFlexLexer' object using the given streams for input 2960 and output. If not specified, the streams default to 'cin' and 2961 'cout', respectively. 'yyFlexLexer' does not take ownership of its 2962 stream arguments. It's up to the user to ensure the streams 2963 pointed to remain alive at least as long as the 'yyFlexLexer' 2964 instance. 2965 2966'virtual int yylex()' 2967 performs the same role is 'yylex()' does for ordinary 'flex' 2968 scanners: it scans the input stream, consuming tokens, until a 2969 rule's action returns a value. If you derive a subclass 'S' from 2970 'yyFlexLexer' and want to access the member functions and variables 2971 of 'S' inside 'yylex()', then you need to use '%option yyclass="S"' 2972 to inform 'flex' that you will be using that subclass instead of 2973 'yyFlexLexer'. In this case, rather than generating 2974 'yyFlexLexer::yylex()', 'flex' generates 'S::yylex()' (and also 2975 generates a dummy 'yyFlexLexer::yylex()' that calls 2976 'yyFlexLexer::LexerError()' if called). 2977 2978'virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0)' 2979'virtual void switch_streams(istream& new_in, ostream& new_out)' 2980 reassigns 'yyin' to 'new_in' (if non-null) and 'yyout' to 'new_out' 2981 (if non-null), deleting the previous input buffer if 'yyin' is 2982 reassigned. 2983 2984'int yylex( istream* new_in, ostream* new_out = 0 )' 2985'int yylex( istream& new_in, ostream& new_out )' 2986 first switches the input streams via 'switch_streams( new_in, 2987 new_out )' and then returns the value of 'yylex()'. 2988 2989 In addition, 'yyFlexLexer' defines the following protected virtual 2990functions which you can redefine in derived classes to tailor the 2991scanner: 2992 2993'virtual int LexerInput( char* buf, int max_size )' 2994 reads up to 'max_size' characters into 'buf' and returns the number 2995 of characters read. To indicate end-of-input, return 0 characters. 2996 Note that 'interactive' scanners (see the '-B' and '-I' flags in 2997 *note Scanner Options::) define the macro 'YY_INTERACTIVE'. If you 2998 redefine 'LexerInput()' and need to take different actions 2999 depending on whether or not the scanner might be scanning an 3000 interactive input source, you can test for the presence of this 3001 name via '#ifdef' statements. 3002 3003'virtual void LexerOutput( const char* buf, int size )' 3004 writes out 'size' characters from the buffer 'buf', which, while 3005 'NUL'-terminated, may also contain internal 'NUL's if the scanner's 3006 rules can match text with 'NUL's in them. 3007 3008'virtual void LexerError( const char* msg )' 3009 reports a fatal error message. The default version of this 3010 function writes the message to the stream 'cerr' and exits. 3011 3012 Note that a 'yyFlexLexer' object contains its _entire_ scanning 3013state. Thus you can use such objects to create reentrant scanners, but 3014see also *note Reentrant::. You can instantiate multiple instances of 3015the same 'yyFlexLexer' class, and you can also combine multiple C++ 3016scanner classes together in the same program using the '-P' option 3017discussed above. 3018 3019 Finally, note that the '%array' feature is not available to C++ 3020scanner classes; you must use '%pointer' (the default). 3021 3022 Here is an example of a simple C++ scanner: 3023 3024 // An example of using the flex C++ scanner class. 3025 3026 %{ 3027 #include <iostream> 3028 using namespace std; 3029 int mylineno = 0; 3030 %} 3031 3032 %option noyywrap c++ 3033 3034 string \"[^\n"]+\" 3035 3036 ws [ \t]+ 3037 3038 alpha [A-Za-z] 3039 dig [0-9] 3040 name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])* 3041 num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)? 3042 num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)? 3043 number {num1}|{num2} 3044 3045 %% 3046 3047 {ws} /* skip blanks and tabs */ 3048 3049 "/*" { 3050 int c; 3051 3052 while((c = yyinput()) != 0) 3053 { 3054 if(c == '\n') 3055 ++mylineno; 3056 3057 else if(c == '*') 3058 { 3059 if((c = yyinput()) == '/') 3060 break; 3061 else 3062 unput(c); 3063 } 3064 } 3065 } 3066 3067 {number} cout << "number " << YYText() << '\n'; 3068 3069 \n mylineno++; 3070 3071 {name} cout << "name " << YYText() << '\n'; 3072 3073 {string} cout << "string " << YYText() << '\n'; 3074 3075 %% 3076 3077 // This include is required if main() is an another source file. 3078 //#include <FlexLexer.h> 3079 3080 int main( int /* argc */, char** /* argv */ ) 3081 { 3082 FlexLexer* lexer = new yyFlexLexer; 3083 while(lexer->yylex() != 0) 3084 ; 3085 return 0; 3086 } 3087 3088 If you want to create multiple (different) lexer classes, you use the 3089'-P' flag (or the 'prefix=' option) to rename each 'yyFlexLexer' to some 3090other 'xxFlexLexer'. You then can include '<FlexLexer.h>' in your other 3091sources once per lexer class, first renaming 'yyFlexLexer' as follows: 3092 3093 #undef yyFlexLexer 3094 #define yyFlexLexer xxFlexLexer 3095 #include <FlexLexer.h> 3096 3097 #undef yyFlexLexer 3098 #define yyFlexLexer zzFlexLexer 3099 #include <FlexLexer.h> 3100 3101 if, for example, you used '%option prefix="xx"' for one of your 3102scanners and '%option prefix="zz"' for the other. 3103 3104 3105File: flex.info, Node: Reentrant, Next: Lex and Posix, Prev: Cxx, Up: Top 3106 310719 Reentrant C Scanners 3108*********************** 3109 3110'flex' has the ability to generate a reentrant C scanner. This is 3111accomplished by specifying '%option reentrant' ('-R') The generated 3112scanner is both portable, and safe to use in one or more separate 3113threads of control. The most common use for reentrant scanners is from 3114within multi-threaded applications. Any thread may create and execute a 3115reentrant 'flex' scanner without the need for synchronization with other 3116threads. 3117 3118* Menu: 3119 3120* Reentrant Uses:: 3121* Reentrant Overview:: 3122* Reentrant Example:: 3123* Reentrant Detail:: 3124* Reentrant Functions:: 3125 3126 3127File: flex.info, Node: Reentrant Uses, Next: Reentrant Overview, Prev: Reentrant, Up: Reentrant 3128 312919.1 Uses for Reentrant Scanners 3130================================ 3131 3132However, there are other uses for a reentrant scanner. For example, you 3133could scan two or more files simultaneously to implement a 'diff' at the 3134token level (i.e., instead of at the character level): 3135 3136 /* Example of maintaining more than one active scanner. */ 3137 3138 do { 3139 int tok1, tok2; 3140 3141 tok1 = yylex( scanner_1 ); 3142 tok2 = yylex( scanner_2 ); 3143 3144 if( tok1 != tok2 ) 3145 printf("Files are different."); 3146 3147 } while ( tok1 && tok2 ); 3148 3149 Another use for a reentrant scanner is recursion. (Note that a 3150recursive scanner can also be created using a non-reentrant scanner and 3151buffer states. *Note Multiple Input Buffers::.) 3152 3153 The following crude scanner supports the 'eval' command by invoking 3154another instance of itself. 3155 3156 /* Example of recursive invocation. */ 3157 3158 %option reentrant 3159 3160 %% 3161 "eval(".+")" { 3162 yyscan_t scanner; 3163 YY_BUFFER_STATE buf; 3164 3165 yylex_init( &scanner ); 3166 yytext[yyleng-1] = ' '; 3167 3168 buf = yy_scan_string( yytext + 5, scanner ); 3169 yylex( scanner ); 3170 3171 yy_delete_buffer(buf,scanner); 3172 yylex_destroy( scanner ); 3173 } 3174 ... 3175 %% 3176 3177 3178File: flex.info, Node: Reentrant Overview, Next: Reentrant Example, Prev: Reentrant Uses, Up: Reentrant 3179 318019.2 An Overview of the Reentrant API 3181===================================== 3182 3183The API for reentrant scanners is different than for non-reentrant 3184scanners. Here is a quick overview of the API: 3185 3186 '%option reentrant' must be specified. 3187 3188 * All functions take one additional argument: 'yyscanner' 3189 3190 * All global variables are replaced by their macro equivalents. (We 3191 tell you this because it may be important to you during debugging.) 3192 3193 * 'yylex_init' and 'yylex_destroy' must be called before and after 3194 'yylex', respectively. 3195 3196 * Accessor methods (get/set functions) provide access to common 3197 'flex' variables. 3198 3199 * User-specific data can be stored in 'yyextra'. 3200 3201 3202File: flex.info, Node: Reentrant Example, Next: Reentrant Detail, Prev: Reentrant Overview, Up: Reentrant 3203 320419.3 Reentrant Example 3205====================== 3206 3207First, an example of a reentrant scanner: 3208 /* This scanner prints "//" comments. */ 3209 3210 %option reentrant stack noyywrap 3211 %x COMMENT 3212 3213 %% 3214 3215 "//" yy_push_state( COMMENT, yyscanner); 3216 .|\n 3217 3218 <COMMENT>\n yy_pop_state( yyscanner ); 3219 <COMMENT>[^\n]+ fprintf( yyout, "%s\n", yytext); 3220 3221 %% 3222 3223 int main ( int argc, char * argv[] ) 3224 { 3225 yyscan_t scanner; 3226 3227 yylex_init ( &scanner ); 3228 yylex ( scanner ); 3229 yylex_destroy ( scanner ); 3230 return 0; 3231 } 3232 3233 3234File: flex.info, Node: Reentrant Detail, Next: Reentrant Functions, Prev: Reentrant Example, Up: Reentrant 3235 323619.4 The Reentrant API in Detail 3237================================ 3238 3239Here are the things you need to do or know to use the reentrant C API of 3240'flex'. 3241 3242* Menu: 3243 3244* Specify Reentrant:: 3245* Extra Reentrant Argument:: 3246* Global Replacement:: 3247* Init and Destroy Functions:: 3248* Accessor Methods:: 3249* Extra Data:: 3250* About yyscan_t:: 3251 3252 3253File: flex.info, Node: Specify Reentrant, Next: Extra Reentrant Argument, Prev: Reentrant Detail, Up: Reentrant Detail 3254 325519.4.1 Declaring a Scanner As Reentrant 3256--------------------------------------- 3257 3258%option reentrant (-reentrant) must be specified. 3259 3260 Notice that '%option reentrant' is specified in the above example 3261(*note Reentrant Example::. Had this option not been specified, 'flex' 3262would have happily generated a non-reentrant scanner without 3263complaining. You may explicitly specify '%option noreentrant', if you 3264do _not_ want a reentrant scanner, although it is not necessary. The 3265default is to generate a non-reentrant scanner. 3266 3267 3268File: flex.info, Node: Extra Reentrant Argument, Next: Global Replacement, Prev: Specify Reentrant, Up: Reentrant Detail 3269 327019.4.2 The Extra Argument 3271------------------------- 3272 3273All functions take one additional argument: 'yyscanner'. 3274 3275 Notice that the calls to 'yy_push_state' and 'yy_pop_state' both have 3276an argument, 'yyscanner' , that is not present in a non-reentrant 3277scanner. Here are the declarations of 'yy_push_state' and 3278'yy_pop_state' in the reentrant scanner: 3279 3280 static void yy_push_state ( int new_state , yyscan_t yyscanner ) ; 3281 static void yy_pop_state ( yyscan_t yyscanner ) ; 3282 3283 Notice that the argument 'yyscanner' appears in the declaration of 3284both functions. In fact, all 'flex' functions in a reentrant scanner 3285have this additional argument. It is always the last argument in the 3286argument list, it is always of type 'yyscan_t' (which is typedef'd to 3287'void *') and it is always named 'yyscanner'. As you may have guessed, 3288'yyscanner' is a pointer to an opaque data structure encapsulating the 3289current state of the scanner. For a list of function declarations, see 3290*note Reentrant Functions::. Note that preprocessor macros, such as 3291'BEGIN', 'ECHO', and 'REJECT', do not take this additional argument. 3292 3293 3294File: flex.info, Node: Global Replacement, Next: Init and Destroy Functions, Prev: Extra Reentrant Argument, Up: Reentrant Detail 3295 329619.4.3 Global Variables Replaced By Macros 3297------------------------------------------ 3298 3299All global variables in traditional flex have been replaced by macro 3300equivalents. 3301 3302 Note that in the above example, 'yyout' and 'yytext' are not plain 3303variables. These are macros that will expand to their equivalent 3304lvalue. All of the familiar 'flex' globals have been replaced by their 3305macro equivalents. In particular, 'yytext', 'yyleng', 'yylineno', 3306'yyin', 'yyout', 'yyextra', 'yylval', and 'yylloc' are macros. You may 3307safely use these macros in actions as if they were plain variables. We 3308only tell you this so you don't expect to link to these variables 3309externally. Currently, each macro expands to a member of an internal 3310struct, e.g., 3311 3312 #define yytext (((struct yyguts_t*)yyscanner)->yytext_r) 3313 3314 One important thing to remember about 'yytext' and friends is that 3315'yytext' is not a global variable in a reentrant scanner, you can not 3316access it directly from outside an action or from other functions. You 3317must use an accessor method, e.g., 'yyget_text', to accomplish this. 3318(See below). 3319 3320 3321File: flex.info, Node: Init and Destroy Functions, Next: Accessor Methods, Prev: Global Replacement, Up: Reentrant Detail 3322 332319.4.4 Init and Destroy Functions 3324--------------------------------- 3325 3326'yylex_init' and 'yylex_destroy' must be called before and after 3327'yylex', respectively. 3328 3329 int yylex_init ( yyscan_t * ptr_yy_globals ) ; 3330 int yylex_init_extra ( YY_EXTRA_TYPE user_defined, yyscan_t * ptr_yy_globals ) ; 3331 int yylex ( yyscan_t yyscanner ) ; 3332 int yylex_destroy ( yyscan_t yyscanner ) ; 3333 3334 The function 'yylex_init' must be called before calling any other 3335function. The argument to 'yylex_init' is the address of an 3336uninitialized pointer to be filled in by 'yylex_init', overwriting any 3337previous contents. The function 'yylex_init_extra' may be used instead, 3338taking as its first argument a variable of type 'YY_EXTRA_TYPE'. See 3339the section on yyextra, below, for more details. 3340 3341 The value stored in 'ptr_yy_globals' should thereafter be passed to 3342'yylex' and 'yylex_destroy'. Flex does not save the argument passed to 3343'yylex_init', so it is safe to pass the address of a local pointer to 3344'yylex_init' so long as it remains in scope for the duration of all 3345calls to the scanner, up to and including the call to 'yylex_destroy'. 3346 3347 The function 'yylex' should be familiar to you by now. The reentrant 3348version takes one argument, which is the value returned (via an 3349argument) by 'yylex_init'. Otherwise, it behaves the same as the 3350non-reentrant version of 'yylex'. 3351 3352 Both 'yylex_init' and 'yylex_init_extra' returns 0 (zero) on success, 3353or non-zero on failure, in which case errno is set to one of the 3354following values: 3355 3356 * ENOMEM Memory allocation error. *Note memory-management::. 3357 * EINVAL Invalid argument. 3358 3359 The function 'yylex_destroy' should be called to free resources used 3360by the scanner. After 'yylex_destroy' is called, the contents of 3361'yyscanner' should not be used. Of course, there is no need to destroy 3362a scanner if you plan to reuse it. A 'flex' scanner (both reentrant and 3363non-reentrant) may be restarted by calling 'yyrestart'. 3364 3365 Below is an example of a program that creates a scanner, uses it, 3366then destroys it when done: 3367 3368 int main () 3369 { 3370 yyscan_t scanner; 3371 int tok; 3372 3373 yylex_init(&scanner); 3374 3375 while ((tok=yylex(scanner)) > 0) 3376 printf("tok=%d yytext=%s\n", tok, yyget_text(scanner)); 3377 3378 yylex_destroy(scanner); 3379 return 0; 3380 } 3381 3382 3383File: flex.info, Node: Accessor Methods, Next: Extra Data, Prev: Init and Destroy Functions, Up: Reentrant Detail 3384 338519.4.5 Accessing Variables with Reentrant Scanners 3386-------------------------------------------------- 3387 3388Accessor methods (get/set functions) provide access to common 'flex' 3389variables. 3390 3391 Many scanners that you build will be part of a larger project. 3392Portions of your project will need access to 'flex' values, such as 3393'yytext'. In a non-reentrant scanner, these values are global, so there 3394is no problem accessing them. However, in a reentrant scanner, there 3395are no global 'flex' values. You can not access them directly. 3396Instead, you must access 'flex' values using accessor methods (get/set 3397functions). Each accessor method is named 'yyget_NAME' or 'yyset_NAME', 3398where 'NAME' is the name of the 'flex' variable you want. For example: 3399 3400 /* Set the last character of yytext to NULL. */ 3401 void chop ( yyscan_t scanner ) 3402 { 3403 int len = yyget_leng( scanner ); 3404 yyget_text( scanner )[len - 1] = '\0'; 3405 } 3406 3407 The above code may be called from within an action like this: 3408 3409 %% 3410 .+\n { chop( yyscanner );} 3411 3412 You may find that '%option header-file' is particularly useful for 3413generating prototypes of all the accessor functions. *Note 3414option-header::. 3415 3416 3417File: flex.info, Node: Extra Data, Next: About yyscan_t, Prev: Accessor Methods, Up: Reentrant Detail 3418 341919.4.6 Extra Data 3420----------------- 3421 3422User-specific data can be stored in 'yyextra'. 3423 3424 In a reentrant scanner, it is unwise to use global variables to 3425communicate with or maintain state between different pieces of your 3426program. However, you may need access to external data or invoke 3427external functions from within the scanner actions. Likewise, you may 3428need to pass information to your scanner (e.g., open file descriptors, 3429or database connections). In a non-reentrant scanner, the only way to 3430do this would be through the use of global variables. 'Flex' allows you 3431to store arbitrary, "extra" data in a scanner. This data is accessible 3432through the accessor methods 'yyget_extra' and 'yyset_extra' from 3433outside the scanner, and through the shortcut macro 'yyextra' from 3434within the scanner itself. They are defined as follows: 3435 3436 #define YY_EXTRA_TYPE void* 3437 YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); 3438 void yyset_extra ( YY_EXTRA_TYPE arbitrary_data , yyscan_t scanner); 3439 3440 In addition, an extra form of 'yylex_init' is provided, 3441'yylex_init_extra'. This function is provided so that the yyextra value 3442can be accessed from within the very first yyalloc, used to allocate the 3443scanner itself. 3444 3445 By default, 'YY_EXTRA_TYPE' is defined as type 'void *'. You may 3446redefine this type using '%option extra-type="your_type"' in the 3447scanner: 3448 3449 /* An example of overriding YY_EXTRA_TYPE. */ 3450 %{ 3451 #include <sys/stat.h> 3452 #include <unistd.h> 3453 %} 3454 %option reentrant 3455 %option extra-type="struct stat *" 3456 %% 3457 3458 __filesize__ printf( "%ld", yyextra->st_size ); 3459 __lastmod__ printf( "%ld", yyextra->st_mtime ); 3460 %% 3461 void scan_file( char* filename ) 3462 { 3463 yyscan_t scanner; 3464 struct stat buf; 3465 FILE *in; 3466 3467 in = fopen( filename, "r" ); 3468 stat( filename, &buf ); 3469 3470 yylex_init_extra( buf, &scanner ); 3471 yyset_in( in, scanner ); 3472 yylex( scanner ); 3473 yylex_destroy( scanner ); 3474 3475 fclose( in ); 3476 } 3477 3478 3479File: flex.info, Node: About yyscan_t, Prev: Extra Data, Up: Reentrant Detail 3480 348119.4.7 About yyscan_t 3482--------------------- 3483 3484'yyscan_t' is defined as: 3485 3486 typedef void* yyscan_t; 3487 3488 It is initialized by 'yylex_init()' to point to an internal 3489structure. You should never access this value directly. In particular, 3490you should never attempt to free it (use 'yylex_destroy()' instead.) 3491 3492 3493File: flex.info, Node: Reentrant Functions, Prev: Reentrant Detail, Up: Reentrant 3494 349519.5 Functions and Macros Available in Reentrant C Scanners 3496=========================================================== 3497 3498The following Functions are available in a reentrant scanner: 3499 3500 char *yyget_text ( yyscan_t scanner ); 3501 int yyget_leng ( yyscan_t scanner ); 3502 FILE *yyget_in ( yyscan_t scanner ); 3503 FILE *yyget_out ( yyscan_t scanner ); 3504 int yyget_lineno ( yyscan_t scanner ); 3505 YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); 3506 int yyget_debug ( yyscan_t scanner ); 3507 3508 void yyset_debug ( int flag, yyscan_t scanner ); 3509 void yyset_in ( FILE * in_str , yyscan_t scanner ); 3510 void yyset_out ( FILE * out_str , yyscan_t scanner ); 3511 void yyset_lineno ( int line_number , yyscan_t scanner ); 3512 void yyset_extra ( YY_EXTRA_TYPE user_defined , yyscan_t scanner ); 3513 3514 There are no "set" functions for yytext and yyleng. This is 3515intentional. 3516 3517 The following Macro shortcuts are available in actions in a reentrant 3518scanner: 3519 3520 yytext 3521 yyleng 3522 yyin 3523 yyout 3524 yylineno 3525 yyextra 3526 yy_flex_debug 3527 3528 In a reentrant C scanner, support for yylineno is always present 3529(i.e., you may access yylineno), but the value is never modified by 3530'flex' unless '%option yylineno' is enabled. This is to allow the user 3531to maintain the line count independently of 'flex'. 3532 3533 The following functions and macros are made available when '%option 3534bison-bridge' ('--bison-bridge') is specified: 3535 3536 YYSTYPE * yyget_lval ( yyscan_t scanner ); 3537 void yyset_lval ( YYSTYPE * yylvalp , yyscan_t scanner ); 3538 yylval 3539 3540 The following functions and macros are made available when '%option 3541bison-locations' ('--bison-locations') is specified: 3542 3543 YYLTYPE *yyget_lloc ( yyscan_t scanner ); 3544 void yyset_lloc ( YYLTYPE * yyllocp , yyscan_t scanner ); 3545 yylloc 3546 3547 Support for yylval assumes that 'YYSTYPE' is a valid type. Support 3548for yylloc assumes that 'YYSLYPE' is a valid type. Typically, these 3549types are generated by 'bison', and are included in section 1 of the 3550'flex' input. 3551 3552 3553File: flex.info, Node: Lex and Posix, Next: Memory Management, Prev: Reentrant, Up: Top 3554 355520 Incompatibilities with Lex and Posix 3556*************************************** 3557 3558'flex' is a rewrite of the AT&T Unix _lex_ tool (the two implementations 3559do not share any code, though), with some extensions and 3560incompatibilities, both of which are of concern to those who wish to 3561write scanners acceptable to both implementations. 'flex' is fully 3562compliant with the POSIX 'lex' specification, except that when using 3563'%pointer' (the default), a call to 'unput()' destroys the contents of 3564'yytext', which is counter to the POSIX specification. In this section 3565we discuss all of the known areas of incompatibility between 'flex', 3566AT&T 'lex', and the POSIX specification. 'flex''s '-l' option turns on 3567maximum compatibility with the original AT&T 'lex' implementation, at 3568the cost of a major loss in the generated scanner's performance. We 3569note below which incompatibilities can be overcome using the '-l' 3570option. 'flex' is fully compatible with 'lex' with the following 3571exceptions: 3572 3573 * The undocumented 'lex' scanner internal variable 'yylineno' is not 3574 supported unless '-l' or '%option yylineno' is used. 3575 3576 * 'yylineno' should be maintained on a per-buffer basis, rather than 3577 a per-scanner (single global variable) basis. 3578 3579 * 'yylineno' is not part of the POSIX specification. 3580 3581 * The 'input()' routine is not redefinable, though it may be called 3582 to read characters following whatever has been matched by a rule. 3583 If 'input()' encounters an end-of-file the normal 'yywrap()' 3584 processing is done. A "real" end-of-file is returned by 'input()' 3585 as 'EOF'. 3586 3587 * Input is instead controlled by defining the 'YY_INPUT()' macro. 3588 3589 * The 'flex' restriction that 'input()' cannot be redefined is in 3590 accordance with the POSIX specification, which simply does not 3591 specify any way of controlling the scanner's input other than by 3592 making an initial assignment to 'yyin'. 3593 3594 * The 'unput()' routine is not redefinable. This restriction is in 3595 accordance with POSIX. 3596 3597 * 'flex' scanners are not as reentrant as 'lex' scanners. In 3598 particular, if you have an interactive scanner and an interrupt 3599 handler which long-jumps out of the scanner, and the scanner is 3600 subsequently called again, you may get the following message: 3601 3602 fatal flex scanner internal error--end of buffer missed 3603 3604 To reenter the scanner, first use: 3605 3606 yyrestart( yyin ); 3607 3608 Note that this call will throw away any buffered input; usually 3609 this isn't a problem with an interactive scanner. *Note 3610 Reentrant::, for 'flex''s reentrant API. 3611 3612 * Also note that 'flex' C++ scanner classes _are_ reentrant, so if 3613 using C++ is an option for you, you should use them instead. *Note 3614 Cxx::, and *note Reentrant:: for details. 3615 3616 * 'output()' is not supported. Output from the ECHO macro is done to 3617 the file-pointer 'yyout' (default 'stdout)'. 3618 3619 * 'output()' is not part of the POSIX specification. 3620 3621 * 'lex' does not support exclusive start conditions (%x), though they 3622 are in the POSIX specification. 3623 3624 * When definitions are expanded, 'flex' encloses them in parentheses. 3625 With 'lex', the following: 3626 3627 NAME [A-Z][A-Z0-9]* 3628 %% 3629 foo{NAME}? printf( "Found it\n" ); 3630 %% 3631 3632 will not match the string 'foo' because when the macro is expanded 3633 the rule is equivalent to 'foo[A-Z][A-Z0-9]*?' and the precedence 3634 is such that the '?' is associated with '[A-Z0-9]*'. With 'flex', 3635 the rule will be expanded to 'foo([A-Z][A-Z0-9]*)?' and so the 3636 string 'foo' will match. 3637 3638 * Note that if the definition begins with '^' or ends with '$' then 3639 it is _not_ expanded with parentheses, to allow these operators to 3640 appear in definitions without losing their special meanings. But 3641 the '<s>', '/', and '<<EOF>>' operators cannot be used in a 'flex' 3642 definition. 3643 3644 * Using '-l' results in the 'lex' behavior of no parentheses around 3645 the definition. 3646 3647 * The POSIX specification is that the definition be enclosed in 3648 parentheses. 3649 3650 * Some implementations of 'lex' allow a rule's action to begin on a 3651 separate line, if the rule's pattern has trailing whitespace: 3652 3653 %% 3654 foo|bar<space here> 3655 { foobar_action();} 3656 3657 'flex' does not support this feature. 3658 3659 * The 'lex' '%r' (generate a Ratfor scanner) option is not supported. 3660 It is not part of the POSIX specification. 3661 3662 * After a call to 'unput()', _yytext_ is undefined until the next 3663 token is matched, unless the scanner was built using '%array'. 3664 This is not the case with 'lex' or the POSIX specification. The 3665 '-l' option does away with this incompatibility. 3666 3667 * The precedence of the '{,}' (numeric range) operator is different. 3668 The AT&T and POSIX specifications of 'lex' interpret 'abc{1,3}' as 3669 match one, two, or three occurrences of 'abc'", whereas 'flex' 3670 interprets it as "match 'ab' followed by one, two, or three 3671 occurrences of 'c'". The '-l' and '--posix' options do away with 3672 this incompatibility. 3673 3674 * The precedence of the '^' operator is different. 'lex' interprets 3675 '^foo|bar' as "match either 'foo' at the beginning of a line, or 3676 'bar' anywhere", whereas 'flex' interprets it as "match either 3677 'foo' or 'bar' if they come at the beginning of a line". The 3678 latter is in agreement with the POSIX specification. 3679 3680 * The special table-size declarations such as '%a' supported by 'lex' 3681 are not required by 'flex' scanners.. 'flex' ignores them. 3682 * The name 'FLEX_SCANNER' is '#define''d so scanners may be written 3683 for use with either 'flex' or 'lex'. Scanners also include 3684 'YY_FLEX_MAJOR_VERSION', 'YY_FLEX_MINOR_VERSION' and 3685 'YY_FLEX_SUBMINOR_VERSION' indicating which version of 'flex' 3686 generated the scanner. For example, for the 2.5.22 release, these 3687 defines would be 2, 5 and 22 respectively. If the version of 3688 'flex' being used is a beta version, then the symbol 'FLEX_BETA' is 3689 defined. 3690 3691 * The symbols '[[' and ']]' in the code sections of the input may 3692 conflict with the m4 delimiters. *Note M4 Dependency::. 3693 3694 The following 'flex' features are not included in 'lex' or the POSIX 3695specification: 3696 3697 * C++ scanners 3698 * %option 3699 * start condition scopes 3700 * start condition stacks 3701 * interactive/non-interactive scanners 3702 * yy_scan_string() and friends 3703 * yyterminate() 3704 * yy_set_interactive() 3705 * yy_set_bol() 3706 * YY_AT_BOL() <<EOF>> 3707 * <*> 3708 * YY_DECL 3709 * YY_START 3710 * YY_USER_ACTION 3711 * YY_USER_INIT 3712 * #line directives 3713 * %{}'s around actions 3714 * reentrant C API 3715 * multiple actions on a line 3716 * almost all of the 'flex' command-line options 3717 3718 The feature "multiple actions on a line" refers to the fact that with 3719'flex' you can put multiple actions on the same line, separated with 3720semi-colons, while with 'lex', the following: 3721 3722 foo handle_foo(); ++num_foos_seen; 3723 3724 is (rather surprisingly) truncated to 3725 3726 foo handle_foo(); 3727 3728 'flex' does not truncate the action. Actions that are not enclosed 3729in braces are simply terminated at the end of the line. 3730 3731 3732File: flex.info, Node: Memory Management, Next: Serialized Tables, Prev: Lex and Posix, Up: Top 3733 373421 Memory Management 3735******************** 3736 3737This chapter describes how flex handles dynamic memory, and how you can 3738override the default behavior. 3739 3740* Menu: 3741 3742* The Default Memory Management:: 3743* Overriding The Default Memory Management:: 3744* A Note About yytext And Memory:: 3745 3746 3747File: flex.info, Node: The Default Memory Management, Next: Overriding The Default Memory Management, Prev: Memory Management, Up: Memory Management 3748 374921.1 The Default Memory Management 3750================================== 3751 3752Flex allocates dynamic memory during initialization, and once in a while 3753from within a call to yylex(). Initialization takes place during the 3754first call to yylex(). Thereafter, flex may reallocate more memory if 3755it needs to enlarge a buffer. As of version 2.5.9 Flex will clean up 3756all memory when you call 'yylex_destroy' *Note faq-memory-leak::. 3757 3758 Flex allocates dynamic memory for four purposes, listed below (1) 3759 376016kB for the input buffer. 3761 Flex allocates memory for the character buffer used to perform 3762 pattern matching. Flex must read ahead from the input stream and 3763 store it in a large character buffer. This buffer is typically the 3764 largest chunk of dynamic memory flex consumes. This buffer will 3765 grow if necessary, doubling the size each time. Flex frees this 3766 memory when you call yylex_destroy(). The default size of this 3767 buffer (16384 bytes) is almost always too large. The ideal size 3768 for this buffer is the length of the longest token expected, in 3769 bytes, plus a little more. Flex will allocate a few extra bytes 3770 for housekeeping. Currently, to override the size of the input 3771 buffer you must '#define YY_BUF_SIZE' to whatever number of bytes 3772 you want. We don't plan to change this in the near future, but we 3773 reserve the right to do so if we ever add a more robust memory 3774 management API. 3775 377664kb for the REJECT state. This will only be allocated if you use REJECT. 3777 The size is large enough to hold the same number of states as 3778 characters in the input buffer. If you override the size of the 3779 input buffer (via 'YY_BUF_SIZE'), then you automatically override 3780 the size of this buffer as well. 3781 3782100 bytes for the start condition stack. 3783 Flex allocates memory for the start condition stack. This is the 3784 stack used for pushing start states, i.e., with yy_push_state(). 3785 It will grow if necessary. Since the states are simply integers, 3786 this stack doesn't consume much memory. This stack is not present 3787 if '%option stack' is not specified. You will rarely need to tune 3788 this buffer. The ideal size for this stack is the maximum depth 3789 expected. The memory for this stack is automatically destroyed 3790 when you call yylex_destroy(). *Note option-stack::. 3791 379240 bytes for each YY_BUFFER_STATE. 3793 Flex allocates memory for each YY_BUFFER_STATE. The buffer state 3794 itself is about 40 bytes, plus an additional large character buffer 3795 (described above.) The initial buffer state is created during 3796 initialization, and with each call to yy_create_buffer(). You 3797 can't tune the size of this, but you can tune the character buffer 3798 as described above. Any buffer state that you explicitly create by 3799 calling yy_create_buffer() is _NOT_ destroyed automatically. You 3800 must call yy_delete_buffer() to free the memory. The exception to 3801 this rule is that flex will delete the current buffer automatically 3802 when you call yylex_destroy(). If you delete the current buffer, 3803 be sure to set it to NULL. That way, flex will not try to delete 3804 the buffer a second time (possibly crashing your program!) At the 3805 time of this writing, flex does not provide a growable stack for 3806 the buffer states. You have to manage that yourself. *Note 3807 Multiple Input Buffers::. 3808 380984 bytes for the reentrant scanner guts 3810 Flex allocates about 84 bytes for the reentrant scanner structure 3811 when you call yylex_init(). It is destroyed when the user calls 3812 yylex_destroy(). 3813 3814 ---------- Footnotes ---------- 3815 3816 (1) The quantities given here are approximate, and may vary due to 3817host architecture, compiler configuration, or due to future enhancements 3818to flex. 3819 3820 3821File: flex.info, Node: Overriding The Default Memory Management, Next: A Note About yytext And Memory, Prev: The Default Memory Management, Up: Memory Management 3822 382321.2 Overriding The Default Memory Management 3824============================================= 3825 3826Flex calls the functions 'yyalloc', 'yyrealloc', and 'yyfree' when it 3827needs to allocate or free memory. By default, these functions are 3828wrappers around the standard C functions, 'malloc', 'realloc', and 3829'free', respectively. You can override the default implementations by 3830telling flex that you will provide your own implementations. 3831 3832 To override the default implementations, you must do two things: 3833 3834 1. Suppress the default implementations by specifying one or more of 3835 the following options: 3836 3837 * '%option noyyalloc' 3838 * '%option noyyrealloc' 3839 * '%option noyyfree'. 3840 3841 2. Provide your own implementation of the following functions: (1) 3842 3843 // For a non-reentrant scanner 3844 void * yyalloc (size_t bytes); 3845 void * yyrealloc (void * ptr, size_t bytes); 3846 void yyfree (void * ptr); 3847 3848 // For a reentrant scanner 3849 void * yyalloc (size_t bytes, void * yyscanner); 3850 void * yyrealloc (void * ptr, size_t bytes, void * yyscanner); 3851 void yyfree (void * ptr, void * yyscanner); 3852 3853 In the following example, we will override all three memory routines. 3854We assume that there is a custom allocator with garbage collection. In 3855order to make this example interesting, we will use a reentrant scanner, 3856passing a pointer to the custom allocator through 'yyextra'. 3857 3858 %{ 3859 #include "some_allocator.h" 3860 %} 3861 3862 /* Suppress the default implementations. */ 3863 %option noyyalloc noyyrealloc noyyfree 3864 %option reentrant 3865 3866 /* Initialize the allocator. */ 3867 %{ 3868 #define YY_EXTRA_TYPE struct allocator* 3869 #define YY_USER_INIT yyextra = allocator_create(); 3870 %} 3871 3872 %% 3873 .|\n ; 3874 %% 3875 3876 /* Provide our own implementations. */ 3877 void * yyalloc (size_t bytes, void* yyscanner) { 3878 return allocator_alloc (yyextra, bytes); 3879 } 3880 3881 void * yyrealloc (void * ptr, size_t bytes, void* yyscanner) { 3882 return allocator_realloc (yyextra, bytes); 3883 } 3884 3885 void yyfree (void * ptr, void * yyscanner) { 3886 /* Do nothing -- we leave it to the garbage collector. */ 3887 } 3888 3889 3890 ---------- Footnotes ---------- 3891 3892 (1) It is not necessary to override all (or any) of the memory 3893management routines. You may, for example, override 'yyrealloc', but 3894not 'yyfree' or 'yyalloc'. 3895 3896 3897File: flex.info, Node: A Note About yytext And Memory, Prev: Overriding The Default Memory Management, Up: Memory Management 3898 389921.3 A Note About yytext And Memory 3900=================================== 3901 3902When flex finds a match, 'yytext' points to the first character of the 3903match in the input buffer. The string itself is part of the input 3904buffer, and is _NOT_ allocated separately. The value of yytext will be 3905overwritten the next time yylex() is called. In short, the value of 3906yytext is only valid from within the matched rule's action. 3907 3908 Often, you want the value of yytext to persist for later processing, 3909i.e., by a parser with non-zero lookahead. In order to preserve yytext, 3910you will have to copy it with strdup() or a similar function. But this 3911introduces some headache because your parser is now responsible for 3912freeing the copy of yytext. If you use a yacc or bison parser, 3913(commonly used with flex), you will discover that the error recovery 3914mechanisms can cause memory to be leaked. 3915 3916 To prevent memory leaks from strdup'd yytext, you will have to track 3917the memory somehow. Our experience has shown that a garbage collection 3918mechanism or a pooled memory mechanism will save you a lot of grief when 3919writing parsers. 3920 3921 3922File: flex.info, Node: Serialized Tables, Next: Diagnostics, Prev: Memory Management, Up: Top 3923 392422 Serialized Tables 3925******************** 3926 3927A 'flex' scanner has the ability to save the DFA tables to a file, and 3928load them at runtime when needed. The motivation for this feature is to 3929reduce the runtime memory footprint. Traditionally, these tables have 3930been compiled into the scanner as C arrays, and are sometimes quite 3931large. Since the tables are compiled into the scanner, the memory used 3932by the tables can never be freed. This is a waste of memory, especially 3933if an application uses several scanners, but none of them at the same 3934time. 3935 3936 The serialization feature allows the tables to be loaded at runtime, 3937before scanning begins. The tables may be discarded when scanning is 3938finished. 3939 3940* Menu: 3941 3942* Creating Serialized Tables:: 3943* Loading and Unloading Serialized Tables:: 3944* Tables File Format:: 3945 3946 3947File: flex.info, Node: Creating Serialized Tables, Next: Loading and Unloading Serialized Tables, Prev: Serialized Tables, Up: Serialized Tables 3948 394922.1 Creating Serialized Tables 3950=============================== 3951 3952You may create a scanner with serialized tables by specifying: 3953 3954 %option tables-file=FILE 3955 or 3956 --tables-file=FILE 3957 3958 These options instruct flex to save the DFA tables to the file FILE. 3959The tables will _not_ be embedded in the generated scanner. The scanner 3960will not function on its own. The scanner will be dependent upon the 3961serialized tables. You must load the tables from this file at runtime 3962before you can scan anything. 3963 3964 If you do not specify a filename to '--tables-file', the tables will 3965be saved to 'lex.yy.tables', where 'yy' is the appropriate prefix. 3966 3967 If your project uses several different scanners, you can concatenate 3968the serialized tables into one file, and flex will find the correct set 3969of tables, using the scanner prefix as part of the lookup key. An 3970example follows: 3971 3972 $ flex --tables-file --prefix=cpp cpp.l 3973 $ flex --tables-file --prefix=c c.l 3974 $ cat lex.cpp.tables lex.c.tables > all.tables 3975 3976 The above example created two scanners, 'cpp', and 'c'. Since we did 3977not specify a filename, the tables were serialized to 'lex.c.tables' and 3978'lex.cpp.tables', respectively. Then, we concatenated the two files 3979together into 'all.tables', which we will distribute with our project. 3980At runtime, we will open the file and tell flex to load the tables from 3981it. Flex will find the correct tables automatically. (See next 3982section). 3983 3984 3985File: flex.info, Node: Loading and Unloading Serialized Tables, Next: Tables File Format, Prev: Creating Serialized Tables, Up: Serialized Tables 3986 398722.2 Loading and Unloading Serialized Tables 3988============================================ 3989 3990If you've built your scanner with '%option tables-file', then you must 3991load the scanner tables at runtime. This can be accomplished with the 3992following function: 3993 3994 -- Function: int yytables_fload (FILE* FP [, yyscan_t SCANNER]) 3995 Locates scanner tables in the stream pointed to by FP and loads 3996 them. Memory for the tables is allocated via 'yyalloc'. You must 3997 call this function before the first call to 'yylex'. The argument 3998 SCANNER only appears in the reentrant scanner. This function 3999 returns '0' (zero) on success, or non-zero on error. 4000 4001 The loaded tables are *not* automatically destroyed (unloaded) when 4002you call 'yylex_destroy'. The reason is that you may create several 4003scanners of the same type (in a reentrant scanner), each of which needs 4004access to these tables. To avoid a nasty memory leak, you must call the 4005following function: 4006 4007 -- Function: int yytables_destroy ([yyscan_t SCANNER]) 4008 Unloads the scanner tables. The tables must be loaded again before 4009 you can scan any more data. The argument SCANNER only appears in 4010 the reentrant scanner. This function returns '0' (zero) on 4011 success, or non-zero on error. 4012 4013 *The functions 'yytables_fload' and 'yytables_destroy' are not 4014thread-safe.* You must ensure that these functions are called exactly 4015once (for each scanner type) in a threaded program, before any thread 4016calls 'yylex'. After the tables are loaded, they are never written to, 4017and no thread protection is required thereafter - until you destroy 4018them. 4019 4020 4021File: flex.info, Node: Tables File Format, Prev: Loading and Unloading Serialized Tables, Up: Serialized Tables 4022 402322.3 Tables File Format 4024======================= 4025 4026This section defines the file format of serialized 'flex' tables. 4027 4028 The tables format allows for one or more sets of tables to be 4029specified, where each set corresponds to a given scanner. Scanners are 4030indexed by name, as described below. The file format is as follows: 4031 4032 TABLE SET 1 4033 +-------------------------------+ 4034 Header | uint32 th_magic; | 4035 | uint32 th_hsize; | 4036 | uint32 th_ssize; | 4037 | uint16 th_flags; | 4038 | char th_version[]; | 4039 | char th_name[]; | 4040 | uint8 th_pad64[]; | 4041 +-------------------------------+ 4042 Table 1 | uint16 td_id; | 4043 | uint16 td_flags; | 4044 | uint32 td_hilen; | 4045 | uint32 td_lolen; | 4046 | void td_data[]; | 4047 | uint8 td_pad64[]; | 4048 +-------------------------------+ 4049 Table 2 | | 4050 . . . 4051 . . . 4052 . . . 4053 . . . 4054 Table n | | 4055 +-------------------------------+ 4056 TABLE SET 2 4057 . 4058 . 4059 . 4060 TABLE SET N 4061 4062 The above diagram shows that a complete set of tables consists of a 4063header followed by multiple individual tables. Furthermore, multiple 4064complete sets may be present in the same file, each set with its own 4065header and tables. The sets are contiguous in the file. The only way 4066to know if another set follows is to check the next four bytes for the 4067magic number (or check for EOF). The header and tables sections are 4068padded to 64-bit boundaries. Below we describe each field in detail. 4069This format does not specify how the scanner will expand the given data, 4070i.e., data may be serialized as int8, but expanded to an int32 array at 4071runtime. This is to reduce the size of the serialized data where 4072possible. Remember, _all integer values are in network byte order_. 4073 4074Fields of a table header: 4075 4076'th_magic' 4077 Magic number, always 0xF13C57B1. 4078 4079'th_hsize' 4080 Size of this entire header, in bytes, including all fields plus any 4081 padding. 4082 4083'th_ssize' 4084 Size of this entire set, in bytes, including the header, all 4085 tables, plus any padding. 4086 4087'th_flags' 4088 Bit flags for this table set. Currently unused. 4089 4090'th_version[]' 4091 Flex version in NULL-terminated string format. e.g., '2.5.13a'. 4092 This is the version of flex that was used to create the serialized 4093 tables. 4094 4095'th_name[]' 4096 Contains the name of this table set. The default is 'yytables', 4097 and is prefixed accordingly, e.g., 'footables'. Must be 4098 NULL-terminated. 4099 4100'th_pad64[]' 4101 Zero or more NULL bytes, padding the entire header to the next 4102 64-bit boundary as calculated from the beginning of the header. 4103 4104Fields of a table: 4105 4106'td_id' 4107 Specifies the table identifier. Possible values are: 4108 'YYTD_ID_ACCEPT (0x01)' 4109 'yy_accept' 4110 'YYTD_ID_BASE (0x02)' 4111 'yy_base' 4112 'YYTD_ID_CHK (0x03)' 4113 'yy_chk' 4114 'YYTD_ID_DEF (0x04)' 4115 'yy_def' 4116 'YYTD_ID_EC (0x05)' 4117 'yy_ec ' 4118 'YYTD_ID_META (0x06)' 4119 'yy_meta' 4120 'YYTD_ID_NUL_TRANS (0x07)' 4121 'yy_NUL_trans' 4122 'YYTD_ID_NXT (0x08)' 4123 'yy_nxt'. This array may be two dimensional. See the 4124 'td_hilen' field below. 4125 'YYTD_ID_RULE_CAN_MATCH_EOL (0x09)' 4126 'yy_rule_can_match_eol' 4127 'YYTD_ID_START_STATE_LIST (0x0A)' 4128 'yy_start_state_list'. This array is handled specially 4129 because it is an array of pointers to structs. See the 4130 'td_flags' field below. 4131 'YYTD_ID_TRANSITION (0x0B)' 4132 'yy_transition'. This array is handled specially because it 4133 is an array of structs. See the 'td_lolen' field below. 4134 'YYTD_ID_ACCLIST (0x0C)' 4135 'yy_acclist' 4136 4137'td_flags' 4138 Bit flags describing how to interpret the data in 'td_data'. The 4139 data arrays are one-dimensional by default, but may be two 4140 dimensional as specified in the 'td_hilen' field. 4141 4142 'YYTD_DATA8 (0x01)' 4143 The data is serialized as an array of type int8. 4144 'YYTD_DATA16 (0x02)' 4145 The data is serialized as an array of type int16. 4146 'YYTD_DATA32 (0x04)' 4147 The data is serialized as an array of type int32. 4148 'YYTD_PTRANS (0x08)' 4149 The data is a list of indexes of entries in the expanded 4150 'yy_transition' array. Each index should be expanded to a 4151 pointer to the corresponding entry in the 'yy_transition' 4152 array. We count on the fact that the 'yy_transition' array 4153 has already been seen. 4154 'YYTD_STRUCT (0x10)' 4155 The data is a list of yy_trans_info structs, each of which 4156 consists of two integers. There is no padding between struct 4157 elements or between structs. The type of each member is 4158 determined by the 'YYTD_DATA*' bits. 4159 4160'td_hilen' 4161 If 'td_hilen' is non-zero, then the data is a two-dimensional 4162 array. Otherwise, the data is a one-dimensional array. 'td_hilen' 4163 contains the number of elements in the higher dimensional array, 4164 and 'td_lolen' contains the number of elements in the lowest 4165 dimension. 4166 4167 Conceptually, 'td_data' is either 'sometype td_data[td_lolen]', or 4168 'sometype td_data[td_hilen][td_lolen]', where 'sometype' is 4169 specified by the 'td_flags' field. It is possible for both 4170 'td_lolen' and 'td_hilen' to be zero, in which case 'td_data' is a 4171 zero length array, and no data is loaded, i.e., this table is 4172 simply skipped. Flex does not currently generate tables of zero 4173 length. 4174 4175'td_lolen' 4176 Specifies the number of elements in the lowest dimension array. If 4177 this is a one-dimensional array, then it is simply the number of 4178 elements in this array. The element size is determined by the 4179 'td_flags' field. 4180 4181'td_data[]' 4182 The table data. This array may be a one- or two-dimensional array, 4183 of type 'int8', 'int16', 'int32', 'struct yy_trans_info', or 4184 'struct yy_trans_info*', depending upon the values in the 4185 'td_flags', 'td_hilen', and 'td_lolen' fields. 4186 4187'td_pad64[]' 4188 Zero or more NULL bytes, padding the entire table to the next 4189 64-bit boundary as calculated from the beginning of this table. 4190 4191 4192File: flex.info, Node: Diagnostics, Next: Limitations, Prev: Serialized Tables, Up: Top 4193 419423 Diagnostics 4195************** 4196 4197The following is a list of 'flex' diagnostic messages: 4198 4199 * 'warning, rule cannot be matched' indicates that the given rule 4200 cannot be matched because it follows other rules that will always 4201 match the same text as it. For example, in the following 'foo' 4202 cannot be matched because it comes after an identifier "catch-all" 4203 rule: 4204 4205 [a-z]+ got_identifier(); 4206 foo got_foo(); 4207 4208 Using 'REJECT' in a scanner suppresses this warning. 4209 4210 * 'warning, -s option given but default rule can be matched' means 4211 that it is possible (perhaps only in a particular start condition) 4212 that the default rule (match any single character) is the only one 4213 that will match a particular input. Since '-s' was given, 4214 presumably this is not intended. 4215 4216 * 'reject_used_but_not_detected undefined' or 4217 'yymore_used_but_not_detected undefined'. These errors can occur 4218 at compile time. They indicate that the scanner uses 'REJECT' or 4219 'yymore()' but that 'flex' failed to notice the fact, meaning that 4220 'flex' scanned the first two sections looking for occurrences of 4221 these actions and failed to find any, but somehow you snuck some in 4222 (via a #include file, for example). Use '%option reject' or 4223 '%option yymore' to indicate to 'flex' that you really do use these 4224 features. 4225 4226 * 'flex scanner jammed'. a scanner compiled with '-s' has 4227 encountered an input string which wasn't matched by any of its 4228 rules. This error can also occur due to internal problems. 4229 4230 * 'token too large, exceeds YYLMAX'. your scanner uses '%array' and 4231 one of its rules matched a string longer than the 'YYLMAX' constant 4232 (8K bytes by default). You can increase the value by #define'ing 4233 'YYLMAX' in the definitions section of your 'flex' input. 4234 4235 * 'scanner requires -8 flag to use the character 'x''. Your scanner 4236 specification includes recognizing the 8-bit character ''x'' and 4237 you did not specify the -8 flag, and your scanner defaulted to 4238 7-bit because you used the '-Cf' or '-CF' table compression 4239 options. See the discussion of the '-7' flag, *note Scanner 4240 Options::, for details. 4241 4242 * 'flex scanner push-back overflow'. you used 'unput()' to push back 4243 so much text that the scanner's buffer could not hold both the 4244 pushed-back text and the current token in 'yytext'. Ideally the 4245 scanner should dynamically resize the buffer in this case, but at 4246 present it does not. 4247 4248 * 'input buffer overflow, can't enlarge buffer because scanner uses 4249 REJECT'. the scanner was working on matching an extremely large 4250 token and needed to expand the input buffer. This doesn't work 4251 with scanners that use 'REJECT'. 4252 4253 * 'fatal flex scanner internal error--end of buffer missed'. This 4254 can occur in a scanner which is reentered after a long-jump has 4255 jumped out (or over) the scanner's activation frame. Before 4256 reentering the scanner, use: 4257 yyrestart( yyin ); 4258 or, as noted above, switch to using the C++ scanner class. 4259 4260 * 'too many start conditions in <> construct!' you listed more start 4261 conditions in a <> construct than exist (so you must have listed at 4262 least one of them twice). 4263 4264 4265File: flex.info, Node: Limitations, Next: Bibliography, Prev: Diagnostics, Up: Top 4266 426724 Limitations 4268************** 4269 4270Some trailing context patterns cannot be properly matched and generate 4271warning messages ('dangerous trailing context'). These are patterns 4272where the ending of the first part of the rule matches the beginning of 4273the second part, such as 'zx*/xy*', where the 'x*' matches the 'x' at 4274the beginning of the trailing context. (Note that the POSIX draft 4275states that the text matched by such patterns is undefined.) For some 4276trailing context rules, parts which are actually fixed-length are not 4277recognized as such, leading to the abovementioned performance loss. In 4278particular, parts using '|' or '{n}' (such as 'foo{3}') are always 4279considered variable-length. Combining trailing context with the special 4280'|' action can result in _fixed_ trailing context being turned into the 4281more expensive _variable_ trailing context. For example, in the 4282following: 4283 4284 %% 4285 abc | 4286 xyz/def 4287 4288 Use of 'unput()' invalidates yytext and yyleng, unless the '%array' 4289directive or the '-l' option has been used. Pattern-matching of 'NUL's 4290is substantially slower than matching other characters. Dynamic 4291resizing of the input buffer is slow, as it entails rescanning all the 4292text matched so far by the current (generally huge) token. Due to both 4293buffering of input and read-ahead, you cannot intermix calls to 4294'<stdio.h>' routines, such as, getchar(), with 'flex' rules and expect 4295it to work. Call 'input()' instead. The total table entries listed by 4296the '-v' flag excludes the number of table entries needed to determine 4297what rule has been matched. The number of entries is equal to the 4298number of DFA states if the scanner does not use 'REJECT', and somewhat 4299greater than the number of states if it does. 'REJECT' cannot be used 4300with the '-f' or '-F' options. 4301 4302 The 'flex' internal algorithms need documentation. 4303 4304 4305File: flex.info, Node: Bibliography, Next: FAQ, Prev: Limitations, Up: Top 4306 430725 Additional Reading 4308********************* 4309 4310You may wish to read more about the following programs: 4311 * lex 4312 * yacc 4313 * sed 4314 * awk 4315 4316 The following books may contain material of interest: 4317 4318 John Levine, Tony Mason, and Doug Brown, _Lex & Yacc_, O'Reilly and 4319Associates. Be sure to get the 2nd edition. 4320 4321 M. E. Lesk and E. Schmidt, _LEX - Lexical Analyzer Generator_ 4322 4323 Alfred Aho, Ravi Sethi and Jeffrey Ullman, _Compilers: Principles, 4324Techniques and Tools_, Addison-Wesley (1986). Describes the 4325pattern-matching techniques used by 'flex' (deterministic finite 4326automata). 4327 4328 4329File: flex.info, Node: FAQ, Next: Appendices, Prev: Bibliography, Up: Top 4330 4331FAQ 4332*** 4333 4334From time to time, the 'flex' maintainer receives certain questions. 4335Rather than repeat answers to well-understood problems, we publish them 4336here. 4337 4338* Menu: 4339 4340* When was flex born?:: 4341* How do I expand backslash-escape sequences in C-style quoted strings?:: 4342* Why do flex scanners call fileno if it is not ANSI compatible?:: 4343* Does flex support recursive pattern definitions?:: 4344* How do I skip huge chunks of input (tens of megabytes) while using flex?:: 4345* Flex is not matching my patterns in the same order that I defined them.:: 4346* My actions are executing out of order or sometimes not at all.:: 4347* How can I have multiple input sources feed into the same scanner at the same time?:: 4348* Can I build nested parsers that work with the same input file?:: 4349* How can I match text only at the end of a file?:: 4350* How can I make REJECT cascade across start condition boundaries?:: 4351* Why cant I use fast or full tables with interactive mode?:: 4352* How much faster is -F or -f than -C?:: 4353* If I have a simple grammar cant I just parse it with flex?:: 4354* Why doesn't yyrestart() set the start state back to INITIAL?:: 4355* How can I match C-style comments?:: 4356* The period isn't working the way I expected.:: 4357* Can I get the flex manual in another format?:: 4358* Does there exist a "faster" NDFA->DFA algorithm?:: 4359* How does flex compile the DFA so quickly?:: 4360* How can I use more than 8192 rules?:: 4361* How do I abandon a file in the middle of a scan and switch to a new file?:: 4362* How do I execute code only during initialization (only before the first scan)?:: 4363* How do I execute code at termination?:: 4364* Where else can I find help?:: 4365* Can I include comments in the "rules" section of the file?:: 4366* I get an error about undefined yywrap().:: 4367* How can I change the matching pattern at run time?:: 4368* How can I expand macros in the input?:: 4369* How can I build a two-pass scanner?:: 4370* How do I match any string not matched in the preceding rules?:: 4371* I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: 4372* Is there a way to make flex treat NULL like a regular character?:: 4373* Whenever flex can not match the input it says "flex scanner jammed".:: 4374* Why doesn't flex have non-greedy operators like perl does?:: 4375* Memory leak - 16386 bytes allocated by malloc.:: 4376* How do I track the byte offset for lseek()?:: 4377* How do I use my own I/O classes in a C++ scanner?:: 4378* How do I skip as many chars as possible?:: 4379* deleteme00:: 4380* Are certain equivalent patterns faster than others?:: 4381* Is backing up a big deal?:: 4382* Can I fake multi-byte character support?:: 4383* deleteme01:: 4384* Can you discuss some flex internals?:: 4385* unput() messes up yy_at_bol:: 4386* The | operator is not doing what I want:: 4387* Why can't flex understand this variable trailing context pattern?:: 4388* The ^ operator isn't working:: 4389* Trailing context is getting confused with trailing optional patterns:: 4390* Is flex GNU or not?:: 4391* ERASEME53:: 4392* I need to scan if-then-else blocks and while loops:: 4393* ERASEME55:: 4394* ERASEME56:: 4395* ERASEME57:: 4396* Is there a repository for flex scanners?:: 4397* How can I conditionally compile or preprocess my flex input file?:: 4398* Where can I find grammars for lex and yacc?:: 4399* I get an end-of-buffer message for each character scanned.:: 4400* unnamed-faq-62:: 4401* unnamed-faq-63:: 4402* unnamed-faq-64:: 4403* unnamed-faq-65:: 4404* unnamed-faq-66:: 4405* unnamed-faq-67:: 4406* unnamed-faq-68:: 4407* unnamed-faq-69:: 4408* unnamed-faq-70:: 4409* unnamed-faq-71:: 4410* unnamed-faq-72:: 4411* unnamed-faq-73:: 4412* unnamed-faq-74:: 4413* unnamed-faq-75:: 4414* unnamed-faq-76:: 4415* unnamed-faq-77:: 4416* unnamed-faq-78:: 4417* unnamed-faq-79:: 4418* unnamed-faq-80:: 4419* unnamed-faq-81:: 4420* unnamed-faq-82:: 4421* unnamed-faq-83:: 4422* unnamed-faq-84:: 4423* unnamed-faq-85:: 4424* unnamed-faq-86:: 4425* unnamed-faq-87:: 4426* unnamed-faq-88:: 4427* unnamed-faq-90:: 4428* unnamed-faq-91:: 4429* unnamed-faq-92:: 4430* unnamed-faq-93:: 4431* unnamed-faq-94:: 4432* unnamed-faq-95:: 4433* unnamed-faq-96:: 4434* unnamed-faq-97:: 4435* unnamed-faq-98:: 4436* unnamed-faq-99:: 4437* unnamed-faq-100:: 4438* unnamed-faq-101:: 4439* What is the difference between YYLEX_PARAM and YY_DECL?:: 4440* Why do I get "conflicting types for yylex" error?:: 4441* How do I access the values set in a Flex action from within a Bison action?:: 4442 4443 4444File: flex.info, Node: When was flex born?, Next: How do I expand backslash-escape sequences in C-style quoted strings?, Up: FAQ 4445 4446When was flex born? 4447=================== 4448 4449Vern Paxson took over the 'Software Tools' lex project from Jef 4450Poskanzer in 1982. At that point it was written in Ratfor. Around 1987 4451or so, Paxson translated it into C, and a legend was born :-). 4452 4453 4454File: flex.info, Node: How do I expand backslash-escape sequences in C-style quoted strings?, Next: Why do flex scanners call fileno if it is not ANSI compatible?, Prev: When was flex born?, Up: FAQ 4455 4456How do I expand backslash-escape sequences in C-style quoted strings? 4457===================================================================== 4458 4459A key point when scanning quoted strings is that you cannot (easily) 4460write a single rule that will precisely match the string if you allow 4461things like embedded escape sequences and newlines. If you try to match 4462strings with a single rule then you'll wind up having to rescan the 4463string anyway to find any escape sequences. 4464 4465 Instead you can use exclusive start conditions and a set of rules, 4466one for matching non-escaped text, one for matching a single escape, one 4467for matching an embedded newline, and one for recognizing the end of the 4468string. Each of these rules is then faced with the question of where to 4469put its intermediary results. The best solution is for the rules to 4470append their local value of 'yytext' to the end of a "string literal" 4471buffer. A rule like the escape-matcher will append to the buffer the 4472meaning of the escape sequence rather than the literal text in 'yytext'. 4473In this way, 'yytext' does not need to be modified at all. 4474 4475 4476File: flex.info, Node: Why do flex scanners call fileno if it is not ANSI compatible?, Next: Does flex support recursive pattern definitions?, Prev: How do I expand backslash-escape sequences in C-style quoted strings?, Up: FAQ 4477 4478Why do flex scanners call fileno if it is not ANSI compatible? 4479============================================================== 4480 4481Flex scanners call 'fileno()' in order to get the file descriptor 4482corresponding to 'yyin'. The file descriptor may be passed to 4483'isatty()' or 'read()', depending upon which '%options' you specified. 4484If your system does not have 'fileno()' support, to get rid of the 4485'read()' call, do not specify '%option read'. To get rid of the 4486'isatty()' call, you must specify one of '%option always-interactive' or 4487'%option never-interactive'. 4488 4489 4490File: flex.info, Node: Does flex support recursive pattern definitions?, Next: How do I skip huge chunks of input (tens of megabytes) while using flex?, Prev: Why do flex scanners call fileno if it is not ANSI compatible?, Up: FAQ 4491 4492Does flex support recursive pattern definitions? 4493================================================ 4494 4495e.g., 4496 4497 %% 4498 block "{"({block}|{statement})*"}" 4499 4500 No. You cannot have recursive definitions. The pattern-matching 4501power of regular expressions in general (and therefore flex scanners, 4502too) is limited. In particular, regular expressions cannot "balance" 4503parentheses to an arbitrary degree. For example, it's impossible to 4504write a regular expression that matches all strings containing the same 4505number of '{'s as '}'s. For more powerful pattern matching, you need a 4506parser, such as 'GNU bison'. 4507 4508 4509File: flex.info, Node: How do I skip huge chunks of input (tens of megabytes) while using flex?, Next: Flex is not matching my patterns in the same order that I defined them., Prev: Does flex support recursive pattern definitions?, Up: FAQ 4510 4511How do I skip huge chunks of input (tens of megabytes) while using flex? 4512======================================================================== 4513 4514Use 'fseek()' (or 'lseek()') to position yyin, then call 'yyrestart()'. 4515 4516 4517File: flex.info, Node: Flex is not matching my patterns in the same order that I defined them., Next: My actions are executing out of order or sometimes not at all., Prev: How do I skip huge chunks of input (tens of megabytes) while using flex?, Up: FAQ 4518 4519Flex is not matching my patterns in the same order that I defined them. 4520======================================================================= 4521 4522'flex' picks the rule that matches the most text (i.e., the longest 4523possible input string). This is because 'flex' uses an entirely 4524different matching technique ("deterministic finite automata") that 4525actually does all of the matching simultaneously, in parallel. (Seems 4526impossible, but it's actually a fairly simple technique once you 4527understand the principles.) 4528 4529 A side-effect of this parallel matching is that when the input 4530matches more than one rule, 'flex' scanners pick the rule that matched 4531the _most_ text. This is explained further in the manual, in the 4532section *Note Matching::. 4533 4534 If you want 'flex' to choose a shorter match, then you can work 4535around this behavior by expanding your short rule to match more text, 4536then put back the extra: 4537 4538 data_.* yyless( 5 ); BEGIN BLOCKIDSTATE; 4539 4540 Another fix would be to make the second rule active only during the 4541'<BLOCKIDSTATE>' start condition, and make that start condition 4542exclusive by declaring it with '%x' instead of '%s'. 4543 4544 A final fix is to change the input language so that the ambiguity for 4545'data_' is removed, by adding characters to it that don't match the 4546identifier rule, or by removing characters (such as '_') from the 4547identifier rule so it no longer matches 'data_'. (Of course, you might 4548also not have the option of changing the input language.) 4549 4550 4551File: flex.info, Node: My actions are executing out of order or sometimes not at all., Next: How can I have multiple input sources feed into the same scanner at the same time?, Prev: Flex is not matching my patterns in the same order that I defined them., Up: FAQ 4552 4553My actions are executing out of order or sometimes not at all. 4554============================================================== 4555 4556Most likely, you have (in error) placed the opening '{' of the action 4557block on a different line than the rule, e.g., 4558 4559 ^(foo|bar) 4560 { <<<--- WRONG! 4561 4562 } 4563 4564 'flex' requires that the opening '{' of an action associated with a 4565rule begin on the same line as does the rule. You need instead to write 4566your rules as follows: 4567 4568 ^(foo|bar) { // CORRECT! 4569 4570 } 4571 4572 4573File: flex.info, Node: How can I have multiple input sources feed into the same scanner at the same time?, Next: Can I build nested parsers that work with the same input file?, Prev: My actions are executing out of order or sometimes not at all., Up: FAQ 4574 4575How can I have multiple input sources feed into the same scanner at the same time? 4576================================================================================== 4577 4578If ... 4579 * your scanner is free of backtracking (verified using 'flex''s '-b' 4580 flag), 4581 * AND you run your scanner interactively ('-I' option; default unless 4582 using special table compression options), 4583 * AND you feed it one character at a time by redefining 'YY_INPUT' to 4584 do so, 4585 4586 then every time it matches a token, it will have exhausted its input 4587buffer (because the scanner is free of backtracking). This means you 4588can safely use 'select()' at the point and only call 'yylex()' for 4589another token if 'select()' indicates there's data available. 4590 4591 That is, move the 'select()' out from the input function to a point 4592where it determines whether 'yylex()' gets called for the next token. 4593 4594 With this approach, you will still have problems if your input can 4595arrive piecemeal; 'select()' could inform you that the beginning of a 4596token is available, you call 'yylex()' to get it, but it winds up 4597blocking waiting for the later characters in the token. 4598 4599 Here's another way: Move your input multiplexing inside of 4600'YY_INPUT'. That is, whenever 'YY_INPUT' is called, it 'select()''s to 4601see where input is available. If input is available for the scanner, it 4602reads and returns the next byte. If input is available from another 4603source, it calls whatever function is responsible for reading from that 4604source. (If no input is available, it blocks until some input is 4605available.) I've used this technique in an interpreter I wrote that 4606both reads keyboard input using a 'flex' scanner and IPC traffic from 4607sockets, and it works fine. 4608 4609 4610File: flex.info, Node: Can I build nested parsers that work with the same input file?, Next: How can I match text only at the end of a file?, Prev: How can I have multiple input sources feed into the same scanner at the same time?, Up: FAQ 4611 4612Can I build nested parsers that work with the same input file? 4613============================================================== 4614 4615This is not going to work without some additional effort. The reason is 4616that 'flex' block-buffers the input it reads from 'yyin'. This means 4617that the "outermost" 'yylex()', when called, will automatically slurp up 4618the first 8K of input available on yyin, and subsequent calls to other 4619'yylex()''s won't see that input. You might be tempted to work around 4620this problem by redefining 'YY_INPUT' to only return a small amount of 4621text, but it turns out that that approach is quite difficult. Instead, 4622the best solution is to combine all of your scanners into one large 4623scanner, using a different exclusive start condition for each. 4624 4625 4626File: flex.info, Node: How can I match text only at the end of a file?, Next: How can I make REJECT cascade across start condition boundaries?, Prev: Can I build nested parsers that work with the same input file?, Up: FAQ 4627 4628How can I match text only at the end of a file? 4629=============================================== 4630 4631There is no way to write a rule which is "match this text, but only if 4632it comes at the end of the file". You can fake it, though, if you 4633happen to have a character lying around that you don't allow in your 4634input. Then you redefine 'YY_INPUT' to call your own routine which, if 4635it sees an 'EOF', returns the magic character first (and remembers to 4636return a real 'EOF' next time it's called). Then you could write: 4637 4638 <COMMENT>(.|\n)*{EOF_CHAR} /* saw comment at EOF */ 4639 4640 4641File: flex.info, Node: How can I make REJECT cascade across start condition boundaries?, Next: Why cant I use fast or full tables with interactive mode?, Prev: How can I match text only at the end of a file?, Up: FAQ 4642 4643How can I make REJECT cascade across start condition boundaries? 4644================================================================ 4645 4646You can do this as follows. Suppose you have a start condition 'A', and 4647after exhausting all of the possible matches in '<A>', you want to try 4648matches in '<INITIAL>'. Then you could use the following: 4649 4650 %x A 4651 %% 4652 <A>rule_that_is_long ...; REJECT; 4653 <A>rule ...; REJECT; /* shorter rule */ 4654 <A>etc. 4655 ... 4656 <A>.|\n { 4657 /* Shortest and last rule in <A>, so 4658 * cascaded REJECTs will eventually 4659 * wind up matching this rule. We want 4660 * to now switch to the initial state 4661 * and try matching from there instead. 4662 */ 4663 yyless(0); /* put back matched text */ 4664 BEGIN(INITIAL); 4665 } 4666 4667 4668File: flex.info, Node: Why cant I use fast or full tables with interactive mode?, Next: How much faster is -F or -f than -C?, Prev: How can I make REJECT cascade across start condition boundaries?, Up: FAQ 4669 4670Why can't I use fast or full tables with interactive mode? 4671========================================================== 4672 4673One of the assumptions flex makes is that interactive applications are 4674inherently slow (they're waiting on a human after all). It has to do 4675with how the scanner detects that it must be finished scanning a token. 4676For interactive scanners, after scanning each character the current 4677state is looked up in a table (essentially) to see whether there's a 4678chance of another input character possibly extending the length of the 4679match. If not, the scanner halts. For non-interactive scanners, the 4680end-of-token test is much simpler, basically a compare with 0, so no 4681memory bus cycles. Since the test occurs in the innermost scanning 4682loop, one would like to make it go as fast as possible. 4683 4684 Still, it seems reasonable to allow the user to choose to trade off a 4685bit of performance in this area to gain the corresponding flexibility. 4686There might be another reason, though, why fast scanners don't support 4687the interactive option. 4688 4689 4690File: flex.info, Node: How much faster is -F or -f than -C?, Next: If I have a simple grammar cant I just parse it with flex?, Prev: Why cant I use fast or full tables with interactive mode?, Up: FAQ 4691 4692How much faster is -F or -f than -C? 4693==================================== 4694 4695Much faster (factor of 2-3). 4696 4697 4698File: flex.info, Node: If I have a simple grammar cant I just parse it with flex?, Next: Why doesn't yyrestart() set the start state back to INITIAL?, Prev: How much faster is -F or -f than -C?, Up: FAQ 4699 4700If I have a simple grammar can't I just parse it with flex? 4701=========================================================== 4702 4703Is your grammar recursive? That's almost always a sign that you're 4704better off using a parser/scanner rather than just trying to use a 4705scanner alone. 4706 4707 4708File: flex.info, Node: Why doesn't yyrestart() set the start state back to INITIAL?, Next: How can I match C-style comments?, Prev: If I have a simple grammar cant I just parse it with flex?, Up: FAQ 4709 4710Why doesn't yyrestart() set the start state back to INITIAL? 4711============================================================ 4712 4713There are two reasons. The first is that there might be programs that 4714rely on the start state not changing across file changes. The second is 4715that beginning with 'flex' version 2.4, use of 'yyrestart()' is no 4716longer required, so fixing the problem there doesn't solve the more 4717general problem. 4718 4719 4720File: flex.info, Node: How can I match C-style comments?, Next: The period isn't working the way I expected., Prev: Why doesn't yyrestart() set the start state back to INITIAL?, Up: FAQ 4721 4722How can I match C-style comments? 4723================================= 4724 4725You might be tempted to try something like this: 4726 4727 "/*".*"*/" // WRONG! 4728 4729 or, worse, this: 4730 4731 "/*"(.|\n)"*/" // WRONG! 4732 4733 The above rules will eat too much input, and blow up on things like: 4734 4735 /* a comment */ do_my_thing( "oops */" ); 4736 4737 Here is one way which allows you to track line information: 4738 4739 <INITIAL>{ 4740 "/*" BEGIN(IN_COMMENT); 4741 } 4742 <IN_COMMENT>{ 4743 "*/" BEGIN(INITIAL); 4744 [^*\n]+ // eat comment in chunks 4745 "*" // eat the lone star 4746 \n yylineno++; 4747 } 4748 4749 4750File: flex.info, Node: The period isn't working the way I expected., Next: Can I get the flex manual in another format?, Prev: How can I match C-style comments?, Up: FAQ 4751 4752The '.' isn't working the way I expected. 4753========================================= 4754 4755Here are some tips for using '.': 4756 4757 * A common mistake is to place the grouping parenthesis AFTER an 4758 operator, when you really meant to place the parenthesis BEFORE the 4759 operator, e.g., you probably want this '(foo|bar)+' and NOT this 4760 '(foo|bar+)'. 4761 4762 The first pattern matches the words 'foo' or 'bar' any number of 4763 times, e.g., it matches the text 'barfoofoobarfoo'. The second 4764 pattern matches a single instance of 'foo' or a single instance of 4765 'bar' followed by one or more 'r's, e.g., it matches the text 4766 'barrrr' . 4767 * A '.' inside '[]''s just means a literal'.' (period), and NOT "any 4768 character except newline". 4769 * Remember that '.' matches any character EXCEPT '\n' (and 'EOF'). 4770 If you really want to match ANY character, including newlines, then 4771 use '(.|\n)' Beware that the regex '(.|\n)+' will match your entire 4772 input! 4773 * Finally, if you want to match a literal '.' (a period), then use 4774 '[.]' or '"."' 4775 4776 4777File: flex.info, Node: Can I get the flex manual in another format?, Next: Does there exist a "faster" NDFA->DFA algorithm?, Prev: The period isn't working the way I expected., Up: FAQ 4778 4779Can I get the flex manual in another format? 4780============================================ 4781 4782The 'flex' source distribution includes a texinfo manual. You are free 4783to convert that texinfo into whatever format you desire. The 'texinfo' 4784package includes tools for conversion to a number of formats. 4785 4786 4787File: flex.info, Node: Does there exist a "faster" NDFA->DFA algorithm?, Next: How does flex compile the DFA so quickly?, Prev: Can I get the flex manual in another format?, Up: FAQ 4788 4789Does there exist a "faster" NDFA->DFA algorithm? 4790================================================ 4791 4792There's no way around the potential exponential running time - it can 4793take you exponential time just to enumerate all of the DFA states. In 4794practice, though, the running time is closer to linear, or sometimes 4795quadratic. 4796 4797 4798File: flex.info, Node: How does flex compile the DFA so quickly?, Next: How can I use more than 8192 rules?, Prev: Does there exist a "faster" NDFA->DFA algorithm?, Up: FAQ 4799 4800How does flex compile the DFA so quickly? 4801========================================= 4802 4803There are two big speed wins that 'flex' uses: 4804 4805 1. It analyzes the input rules to construct equivalence classes for 4806 those characters that always make the same transitions. It then 4807 rewrites the NFA using equivalence classes for transitions instead 4808 of characters. This cuts down the NFA->DFA computation time 4809 dramatically, to the point where, for uncompressed DFA tables, the 4810 DFA generation is often I/O bound in writing out the tables. 4811 2. It maintains hash values for previously computed DFA states, so 4812 testing whether a newly constructed DFA state is equivalent to a 4813 previously constructed state can be done very quickly, by first 4814 comparing hash values. 4815 4816 4817File: flex.info, Node: How can I use more than 8192 rules?, Next: How do I abandon a file in the middle of a scan and switch to a new file?, Prev: How does flex compile the DFA so quickly?, Up: FAQ 4818 4819How can I use more than 8192 rules? 4820=================================== 4821 4822'Flex' is compiled with an upper limit of 8192 rules per scanner. If 4823you need more than 8192 rules in your scanner, you'll have to recompile 4824'flex' with the following changes in 'flexdef.h': 4825 4826 < #define YY_TRAILING_MASK 0x2000 4827 < #define YY_TRAILING_HEAD_MASK 0x4000 4828 -- 4829 > #define YY_TRAILING_MASK 0x20000000 4830 > #define YY_TRAILING_HEAD_MASK 0x40000000 4831 4832 This should work okay as long as your C compiler uses 32 bit 4833integers. But you might want to think about whether using such a huge 4834number of rules is the best way to solve your problem. 4835 4836 The following may also be relevant: 4837 4838 With luck, you should be able to increase the definitions in 4839flexdef.h for: 4840 4841 #define JAMSTATE -32766 /* marks a reference to the state that always jams */ 4842 #define MAXIMUM_MNS 31999 4843 #define BAD_SUBSCRIPT -32767 4844 4845 recompile everything, and it'll all work. Flex only has these 484616-bit-like values built into it because a long time ago it was 4847developed on a machine with 16-bit ints. I've given this advice to 4848others in the past but haven't heard back from them whether it worked 4849okay or not... 4850 4851 4852File: flex.info, Node: How do I abandon a file in the middle of a scan and switch to a new file?, Next: How do I execute code only during initialization (only before the first scan)?, Prev: How can I use more than 8192 rules?, Up: FAQ 4853 4854How do I abandon a file in the middle of a scan and switch to a new file? 4855========================================================================= 4856 4857Just call 'yyrestart(newfile)'. Be sure to reset the start state if you 4858want a "fresh start, since 'yyrestart' does NOT reset the start state 4859back to 'INITIAL'. 4860 4861 4862File: flex.info, Node: How do I execute code only during initialization (only before the first scan)?, Next: How do I execute code at termination?, Prev: How do I abandon a file in the middle of a scan and switch to a new file?, Up: FAQ 4863 4864How do I execute code only during initialization (only before the first scan)? 4865============================================================================== 4866 4867You can specify an initial action by defining the macro 'YY_USER_INIT' 4868(though note that 'yyout' may not be available at the time this macro is 4869executed). Or you can add to the beginning of your rules section: 4870 4871 %% 4872 /* Must be indented! */ 4873 static int did_init = 0; 4874 4875 if ( ! did_init ){ 4876 do_my_init(); 4877 did_init = 1; 4878 } 4879 4880 4881File: flex.info, Node: How do I execute code at termination?, Next: Where else can I find help?, Prev: How do I execute code only during initialization (only before the first scan)?, Up: FAQ 4882 4883How do I execute code at termination? 4884===================================== 4885 4886You can specify an action for the '<<EOF>>' rule. 4887 4888 4889File: flex.info, Node: Where else can I find help?, Next: Can I include comments in the "rules" section of the file?, Prev: How do I execute code at termination?, Up: FAQ 4890 4891Where else can I find help? 4892=========================== 4893 4894You can find the flex homepage on the web at 4895<http://flex.sourceforge.net/>. See that page for details about flex 4896mailing lists as well. 4897 4898 4899File: flex.info, Node: Can I include comments in the "rules" section of the file?, Next: I get an error about undefined yywrap()., Prev: Where else can I find help?, Up: FAQ 4900 4901Can I include comments in the "rules" section of the file? 4902========================================================== 4903 4904Yes, just about anywhere you want to. See the manual for the specific 4905syntax. 4906 4907 4908File: flex.info, Node: I get an error about undefined yywrap()., Next: How can I change the matching pattern at run time?, Prev: Can I include comments in the "rules" section of the file?, Up: FAQ 4909 4910I get an error about undefined yywrap(). 4911======================================== 4912 4913You must supply a 'yywrap()' function of your own, or link to 'libfl.a' 4914(which provides one), or use 4915 4916 %option noyywrap 4917 4918 in your source to say you don't want a 'yywrap()' function. 4919 4920 4921File: flex.info, Node: How can I change the matching pattern at run time?, Next: How can I expand macros in the input?, Prev: I get an error about undefined yywrap()., Up: FAQ 4922 4923How can I change the matching pattern at run time? 4924================================================== 4925 4926You can't, it's compiled into a static table when flex builds the 4927scanner. 4928 4929 4930File: flex.info, Node: How can I expand macros in the input?, Next: How can I build a two-pass scanner?, Prev: How can I change the matching pattern at run time?, Up: FAQ 4931 4932How can I expand macros in the input? 4933===================================== 4934 4935The best way to approach this problem is at a higher level, e.g., in the 4936parser. 4937 4938 However, you can do this using multiple input buffers. 4939 4940 %% 4941 macro/[a-z]+ { 4942 /* Saw the macro "macro" followed by extra stuff. */ 4943 main_buffer = YY_CURRENT_BUFFER; 4944 expansion_buffer = yy_scan_string(expand(yytext)); 4945 yy_switch_to_buffer(expansion_buffer); 4946 } 4947 4948 <<EOF>> { 4949 if ( expansion_buffer ) 4950 { 4951 // We were doing an expansion, return to where 4952 // we were. 4953 yy_switch_to_buffer(main_buffer); 4954 yy_delete_buffer(expansion_buffer); 4955 expansion_buffer = 0; 4956 } 4957 else 4958 yyterminate(); 4959 } 4960 4961 You probably will want a stack of expansion buffers to allow nested 4962macros. From the above though hopefully the idea is clear. 4963 4964 4965File: flex.info, Node: How can I build a two-pass scanner?, Next: How do I match any string not matched in the preceding rules?, Prev: How can I expand macros in the input?, Up: FAQ 4966 4967How can I build a two-pass scanner? 4968=================================== 4969 4970One way to do it is to filter the first pass to a temporary file, then 4971process the temporary file on the second pass. You will probably see a 4972performance hit, due to all the disk I/O. 4973 4974 When you need to look ahead far forward like this, it almost always 4975means that the right solution is to build a parse tree of the entire 4976input, then walk it after the parse in order to generate the output. In 4977a sense, this is a two-pass approach, once through the text and once 4978through the parse tree, but the performance hit for the latter is 4979usually an order of magnitude smaller, since everything is already 4980classified, in binary format, and residing in memory. 4981 4982 4983File: flex.info, Node: How do I match any string not matched in the preceding rules?, Next: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Prev: How can I build a two-pass scanner?, Up: FAQ 4984 4985How do I match any string not matched in the preceding rules? 4986============================================================= 4987 4988One way to assign precedence, is to place the more specific rules first. 4989If two rules would match the same input (same sequence of characters) 4990then the first rule listed in the 'flex' input wins, e.g., 4991 4992 %% 4993 foo[a-zA-Z_]+ return FOO_ID; 4994 bar[a-zA-Z_]+ return BAR_ID; 4995 [a-zA-Z_]+ return GENERIC_ID; 4996 4997 Note that the rule '[a-zA-Z_]+' must come *after* the others. It 4998will match the same amount of text as the more specific rules, and in 4999that case the 'flex' scanner will pick the first rule listed in your 5000scanner as the one to match. 5001 5002 5003File: flex.info, Node: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Next: Is there a way to make flex treat NULL like a regular character?, Prev: How do I match any string not matched in the preceding rules?, Up: FAQ 5004 5005I am trying to port code from AT&T lex that uses yysptr and yysbuf. 5006=================================================================== 5007 5008Those are internal variables pointing into the AT&T scanner's input 5009buffer. I imagine they're being manipulated in user versions of the 5010'input()' and 'unput()' functions. If so, what you need to do is 5011analyze those functions to figure out what they're doing, and then 5012replace 'input()' with an appropriate definition of 'YY_INPUT'. You 5013shouldn't need to (and must not) replace 'flex''s 'unput()' function. 5014 5015 5016File: flex.info, Node: Is there a way to make flex treat NULL like a regular character?, Next: Whenever flex can not match the input it says "flex scanner jammed"., Prev: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Up: FAQ 5017 5018Is there a way to make flex treat NULL like a regular character? 5019================================================================ 5020 5021Yes, '\0' and '\x00' should both do the trick. Perhaps you have an 5022ancient version of 'flex'. The latest release is version 2.6.0. 5023 5024 5025File: flex.info, Node: Whenever flex can not match the input it says "flex scanner jammed"., Next: Why doesn't flex have non-greedy operators like perl does?, Prev: Is there a way to make flex treat NULL like a regular character?, Up: FAQ 5026 5027Whenever flex can not match the input it says "flex scanner jammed". 5028==================================================================== 5029 5030You need to add a rule that matches the otherwise-unmatched text, e.g., 5031 5032 %option yylineno 5033 %% 5034 [[a bunch of rules here]] 5035 5036 . printf("bad input character '%s' at line %d\n", yytext, yylineno); 5037 5038 See '%option default' for more information. 5039 5040 5041File: flex.info, Node: Why doesn't flex have non-greedy operators like perl does?, Next: Memory leak - 16386 bytes allocated by malloc., Prev: Whenever flex can not match the input it says "flex scanner jammed"., Up: FAQ 5042 5043Why doesn't flex have non-greedy operators like perl does? 5044========================================================== 5045 5046A DFA can do a non-greedy match by stopping the first time it enters an 5047accepting state, instead of consuming input until it determines that no 5048further matching is possible (a "jam" state). This is actually easier 5049to implement than longest leftmost match (which flex does). 5050 5051 But it's also much less useful than longest leftmost match. In 5052general, when you find yourself wishing for non-greedy matching, that's 5053usually a sign that you're trying to make the scanner do some parsing. 5054That's generally the wrong approach, since it lacks the power to do a 5055decent job. Better is to either introduce a separate parser, or to 5056split the scanner into multiple scanners using (exclusive) start 5057conditions. 5058 5059 You might have a separate start state once you've seen the 'BEGIN'. 5060In that state, you might then have a regex that will match 'END' (to 5061kick you out of the state), and perhaps '(.|\n)' to get a single 5062character within the chunk ... 5063 5064 This approach also has much better error-reporting properties. 5065 5066 5067File: flex.info, Node: Memory leak - 16386 bytes allocated by malloc., Next: How do I track the byte offset for lseek()?, Prev: Why doesn't flex have non-greedy operators like perl does?, Up: FAQ 5068 5069Memory leak - 16386 bytes allocated by malloc. 5070============================================== 5071 5072UPDATED 2002-07-10: As of 'flex' version 2.5.9, this leak means that you 5073did not call 'yylex_destroy()'. If you are using an earlier version of 5074'flex', then read on. 5075 5076 The leak is about 16426 bytes. That is, (8192 * 2 + 2) for the 5077read-buffer, and about 40 for 'struct yy_buffer_state' (depending upon 5078alignment). The leak is in the non-reentrant C scanner only (NOT in the 5079reentrant scanner, NOT in the C++ scanner). Since 'flex' doesn't know 5080when you are done, the buffer is never freed. 5081 5082 However, the leak won't multiply since the buffer is reused no matter 5083how many times you call 'yylex()'. 5084 5085 If you want to reclaim the memory when you are completely done 5086scanning, then you might try this: 5087 5088 /* For non-reentrant C scanner only. */ 5089 yy_delete_buffer(YY_CURRENT_BUFFER); 5090 yy_init = 1; 5091 5092 Note: 'yy_init' is an "internal variable", and hasn't been tested in 5093this situation. It is possible that some other globals may need 5094resetting as well. 5095 5096 5097File: flex.info, Node: How do I track the byte offset for lseek()?, Next: How do I use my own I/O classes in a C++ scanner?, Prev: Memory leak - 16386 bytes allocated by malloc., Up: FAQ 5098 5099How do I track the byte offset for lseek()? 5100=========================================== 5101 5102 > We thought that it would be possible to have this number through the 5103 > evaluation of the following expression: 5104 > 5105 > seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf 5106 5107 While this is the right idea, it has two problems. The first is that 5108it's possible that 'flex' will request less than 'YY_READ_BUF_SIZE' 5109during an invocation of 'YY_INPUT' (or that your input source will 5110return less even though 'YY_READ_BUF_SIZE' bytes were requested). The 5111second problem is that when refilling its internal buffer, 'flex' keeps 5112some characters from the previous buffer (because usually it's in the 5113middle of a match, and needs those characters to construct 'yytext' for 5114the match once it's done). Because of this, 'yy_c_buf_p - 5115YY_CURRENT_BUFFER->yy_ch_buf' won't be exactly the number of characters 5116already read from the current buffer. 5117 5118 An alternative solution is to count the number of characters you've 5119matched since starting to scan. This can be done by using 5120'YY_USER_ACTION'. For example, 5121 5122 #define YY_USER_ACTION num_chars += yyleng; 5123 5124 (You need to be careful to update your bookkeeping if you use 5125'yymore('), 'yyless()', 'unput()', or 'input()'.) 5126 5127 5128File: flex.info, Node: How do I use my own I/O classes in a C++ scanner?, Next: How do I skip as many chars as possible?, Prev: How do I track the byte offset for lseek()?, Up: FAQ 5129 5130How do I use my own I/O classes in a C++ scanner? 5131================================================= 5132 5133When the flex C++ scanning class rewrite finally happens, then this sort 5134of thing should become much easier. 5135 5136 You can do this by passing the various functions (such as 5137'LexerInput()' and 'LexerOutput()') NULL 'iostream*''s, and then dealing 5138with your own I/O classes surreptitiously (i.e., stashing them in 5139special member variables). This works because the only assumption about 5140the lexer regarding what's done with the iostream's is that they're 5141ultimately passed to 'LexerInput()' and 'LexerOutput', which then do 5142whatever is necessary with them. 5143 5144 5145File: flex.info, Node: How do I skip as many chars as possible?, Next: deleteme00, Prev: How do I use my own I/O classes in a C++ scanner?, Up: FAQ 5146 5147How do I skip as many chars as possible? 5148======================================== 5149 5150How do I skip as many chars as possible - without interfering with the 5151other patterns? 5152 5153 In the example below, we want to skip over characters until we see 5154the phrase "endskip". The following will _NOT_ work correctly (do you 5155see why not?) 5156 5157 /* INCORRECT SCANNER */ 5158 %x SKIP 5159 %% 5160 <INITIAL>startskip BEGIN(SKIP); 5161 ... 5162 <SKIP>"endskip" BEGIN(INITIAL); 5163 <SKIP>.* ; 5164 5165 The problem is that the pattern .* will eat up the word "endskip." 5166The simplest (but slow) fix is: 5167 5168 <SKIP>"endskip" BEGIN(INITIAL); 5169 <SKIP>. ; 5170 5171 The fix involves making the second rule match more, without making it 5172match "endskip" plus something else. So for example: 5173 5174 <SKIP>"endskip" BEGIN(INITIAL); 5175 <SKIP>[^e]+ ; 5176 <SKIP>. ;/* so you eat up e's, too */ 5177 5178 5179File: flex.info, Node: deleteme00, Next: Are certain equivalent patterns faster than others?, Prev: How do I skip as many chars as possible?, Up: FAQ 5180 5181deleteme00 5182========== 5183 5184 QUESTION: 5185 When was flex born? 5186 5187 Vern Paxson took over 5188 the Software Tools lex project from Jef Poskanzer in 1982. At that point it 5189 was written in Ratfor. Around 1987 or so, Paxson translated it into C, and 5190 a legend was born :-). 5191 5192 5193File: flex.info, Node: Are certain equivalent patterns faster than others?, Next: Is backing up a big deal?, Prev: deleteme00, Up: FAQ 5194 5195Are certain equivalent patterns faster than others? 5196=================================================== 5197 5198 To: Adoram Rogel <adoram@orna.hybridge.com> 5199 Subject: Re: Flex 2.5.2 performance questions 5200 In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT. 5201 Date: Wed, 18 Sep 96 10:51:02 PDT 5202 From: Vern Paxson <vern> 5203 5204 [Note, the most recent flex release is 2.5.4, which you can get from 5205 ftp.ee.lbl.gov. It has bug fixes over 2.5.2 and 2.5.3.] 5206 5207 > 1. Using the pattern 5208 > ([Ff](oot)?)?[Nn](ote)?(\.)? 5209 > instead of 5210 > (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.))) 5211 > (in a very complicated flex program) caused the program to slow from 5212 > 300K+/min to 100K/min (no other changes were done). 5213 5214 These two are not equivalent. For example, the first can match "footnote." 5215 but the second can only match "footnote". This is almost certainly the 5216 cause in the discrepancy - the slower scanner run is matching more tokens, 5217 and/or having to do more backing up. 5218 5219 > 2. Which of these two are better: [Ff]oot or (F|f)oot ? 5220 5221 From a performance point of view, they're equivalent (modulo presumably 5222 minor effects such as memory cache hit rates; and the presence of trailing 5223 context, see below). From a space point of view, the first is slightly 5224 preferable. 5225 5226 > 3. I have a pattern that look like this: 5227 > pats {p1}|{p2}|{p3}|...|{p50} (50 patterns ORd) 5228 > 5229 > running yet another complicated program that includes the following rule: 5230 > <snext>{and}/{no4}{bb}{pats} 5231 > 5232 > gets me to "too complicated - over 32,000 states"... 5233 5234 I can't tell from this example whether the trailing context is variable-length 5235 or fixed-length (it could be the latter if {and} is fixed-length). If it's 5236 variable length, which flex -p will tell you, then this reflects a basic 5237 performance problem, and if you can eliminate it by restructuring your 5238 scanner, you will see significant improvement. 5239 5240 > so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about 5241 > 10 patterns and changed the rule to be 5 rules. 5242 > This did compile, but what is the rule of thumb here ? 5243 5244 The rule is to avoid trailing context other than fixed-length, in which for 5245 a/b, either the 'a' pattern or the 'b' pattern have a fixed length. Use 5246 of the '|' operator automatically makes the pattern variable length, so in 5247 this case '[Ff]oot' is preferred to '(F|f)oot'. 5248 5249 > 4. I changed a rule that looked like this: 5250 > <snext8>{and}{bb}/{ROMAN}[^A-Za-z] { BEGIN... 5251 > 5252 > to the next 2 rules: 5253 > <snext8>{and}{bb}/{ROMAN}[A-Za-z] { ECHO;} 5254 > <snext8>{and}{bb}/{ROMAN} { BEGIN... 5255 > 5256 > Again, I understand the using [^...] will cause a great performance loss 5257 5258 Actually, it doesn't cause any sort of performance loss. It's a surprising 5259 fact about regular expressions that they always match in linear time 5260 regardless of how complex they are. 5261 5262 > but are there any specific rules about it ? 5263 5264 See the "Performance Considerations" section of the man page, and also 5265 the example in MISC/fastwc/. 5266 5267 Vern 5268 5269 5270File: flex.info, Node: Is backing up a big deal?, Next: Can I fake multi-byte character support?, Prev: Are certain equivalent patterns faster than others?, Up: FAQ 5271 5272Is backing up a big deal? 5273========================= 5274 5275 To: Adoram Rogel <adoram@hybridge.com> 5276 Subject: Re: Flex 2.5.2 performance questions 5277 In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT. 5278 Date: Thu, 19 Sep 96 09:58:00 PDT 5279 From: Vern Paxson <vern> 5280 5281 > a lot about the backing up problem. 5282 > I believe that there lies my biggest problem, and I'll try to improve 5283 > it. 5284 5285 Since you have variable trailing context, this is a bigger performance 5286 problem. Fixing it is usually easier than fixing backing up, which in a 5287 complicated scanner (yours seems to fit the bill) can be extremely 5288 difficult to do correctly. 5289 5290 You also don't mention what flags you are using for your scanner. 5291 -f makes a large speed difference, and -Cfe buys you nearly as much 5292 speed but the resulting scanner is considerably smaller. 5293 5294 > I have an | operator in {and} and in {pats} so both of them are variable 5295 > length. 5296 5297 -p should have reported this. 5298 5299 > Is changing one of them to fixed-length is enough ? 5300 5301 Yes. 5302 5303 > Is it possible to change the 32,000 states limit ? 5304 5305 Yes. I've appended instructions on how. Before you make this change, 5306 though, you should think about whether there are ways to fundamentally 5307 simplify your scanner - those are certainly preferable! 5308 5309 Vern 5310 5311 To increase the 32K limit (on a machine with 32 bit integers), you increase 5312 the magnitude of the following in flexdef.h: 5313 5314 #define JAMSTATE -32766 /* marks a reference to the state that always jams */ 5315 #define MAXIMUM_MNS 31999 5316 #define BAD_SUBSCRIPT -32767 5317 #define MAX_SHORT 32700 5318 5319 Adding a 0 or two after each should do the trick. 5320 5321 5322File: flex.info, Node: Can I fake multi-byte character support?, Next: deleteme01, Prev: Is backing up a big deal?, Up: FAQ 5323 5324Can I fake multi-byte character support? 5325======================================== 5326 5327 To: Heeman_Lee@hp.com 5328 Subject: Re: flex - multi-byte support? 5329 In-reply-to: Your message of Thu, 03 Oct 1996 17:24:04 PDT. 5330 Date: Fri, 04 Oct 1996 11:42:18 PDT 5331 From: Vern Paxson <vern> 5332 5333 > I assume as long as my *.l file defines the 5334 > range of expected character code values (in octal format), flex will 5335 > scan the file and read multi-byte characters correctly. But I have no 5336 > confidence in this assumption. 5337 5338 Your lack of confidence is justified - this won't work. 5339 5340 Flex has in it a widespread assumption that the input is processed 5341 one byte at a time. Fixing this is on the to-do list, but is involved, 5342 so it won't happen any time soon. In the interim, the best I can suggest 5343 (unless you want to try fixing it yourself) is to write your rules in 5344 terms of pairs of bytes, using definitions in the first section: 5345 5346 X \xfe\xc2 5347 ... 5348 %% 5349 foo{X}bar found_foo_fe_c2_bar(); 5350 5351 etc. Definitely a pain - sorry about that. 5352 5353 By the way, the email address you used for me is ancient, indicating you 5354 have a very old version of flex. You can get the most recent, 2.5.4, from 5355 ftp.ee.lbl.gov. 5356 5357 Vern 5358 5359 5360File: flex.info, Node: deleteme01, Next: Can you discuss some flex internals?, Prev: Can I fake multi-byte character support?, Up: FAQ 5361 5362deleteme01 5363========== 5364 5365 To: moleary@primus.com 5366 Subject: Re: Flex / Unicode compatibility question 5367 In-reply-to: Your message of Tue, 22 Oct 1996 10:15:42 PDT. 5368 Date: Tue, 22 Oct 1996 11:06:13 PDT 5369 From: Vern Paxson <vern> 5370 5371 Unfortunately flex at the moment has a widespread assumption within it 5372 that characters are processed 8 bits at a time. I don't see any easy 5373 fix for this (other than writing your rules in terms of double characters - 5374 a pain). I also don't know of a wider lex, though you might try surfing 5375 the Plan 9 stuff because I know it's a Unicode system, and also the PCCT 5376 toolkit (try searching say Alta Vista for "Purdue Compiler Construction 5377 Toolkit"). 5378 5379 Fixing flex to handle wider characters is on the long-term to-do list. 5380 But since flex is a strictly spare-time project these days, this probably 5381 won't happen for quite a while, unless someone else does it first. 5382 5383 Vern 5384 5385 5386File: flex.info, Node: Can you discuss some flex internals?, Next: unput() messes up yy_at_bol, Prev: deleteme01, Up: FAQ 5387 5388Can you discuss some flex internals? 5389==================================== 5390 5391 To: Johan Linde <jl@theophys.kth.se> 5392 Subject: Re: translation of flex 5393 In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST. 5394 Date: Mon, 11 Nov 1996 10:33:50 PST 5395 From: Vern Paxson <vern> 5396 5397 > I'm working for the Swedish team translating GNU program, and I'm currently 5398 > working with flex. I have a few questions about some of the messages which 5399 > I hope you can answer. 5400 5401 All of the things you're wondering about, by the way, concerning flex 5402 internals - probably the only person who understands what they mean in 5403 English is me! So I wouldn't worry too much about getting them right. 5404 That said ... 5405 5406 > #: main.c:545 5407 > msgid " %d protos created\n" 5408 > 5409 > Does proto mean prototype? 5410 5411 Yes - prototypes of state compression tables. 5412 5413 > #: main.c:539 5414 > msgid " %d/%d (peak %d) template nxt-chk entries created\n" 5415 > 5416 > Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?) 5417 > However, 'template next-check entries' doesn't make much sense to me. To be 5418 > able to find a good translation I need to know a little bit more about it. 5419 5420 There is a scheme in the Aho/Sethi/Ullman compiler book for compressing 5421 scanner tables. It involves creating two pairs of tables. The first has 5422 "base" and "default" entries, the second has "next" and "check" entries. 5423 The "base" entry is indexed by the current state and yields an index into 5424 the next/check table. The "default" entry gives what to do if the state 5425 transition isn't found in next/check. The "next" entry gives the next 5426 state to enter, but only if the "check" entry verifies that this entry is 5427 correct for the current state. Flex creates templates of series of 5428 next/check entries and then encodes differences from these templates as a 5429 way to compress the tables. 5430 5431 > #: main.c:533 5432 > msgid " %d/%d base-def entries created\n" 5433 > 5434 > The same problem here for 'base-def'. 5435 5436 See above. 5437 5438 Vern 5439 5440 5441File: flex.info, Node: unput() messes up yy_at_bol, Next: The | operator is not doing what I want, Prev: Can you discuss some flex internals?, Up: FAQ 5442 5443unput() messes up yy_at_bol 5444=========================== 5445 5446 To: Xinying Li <xli@npac.syr.edu> 5447 Subject: Re: FLEX ? 5448 In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST. 5449 Date: Wed, 13 Nov 1996 19:51:54 PST 5450 From: Vern Paxson <vern> 5451 5452 > "unput()" them to input flow, question occurs. If I do this after I scan 5453 > a carriage, the variable "YY_CURRENT_BUFFER->yy_at_bol" is changed. That 5454 > means the carriage flag has gone. 5455 5456 You can control this by calling yy_set_bol(). It's described in the manual. 5457 5458 > And if in pre-reading it goes to the end of file, is anything done 5459 > to control the end of curren buffer and end of file? 5460 5461 No, there's no way to put back an end-of-file. 5462 5463 > By the way I am using flex 2.5.2 and using the "-l". 5464 5465 The latest release is 2.5.4, by the way. It fixes some bugs in 2.5.2 and 5466 2.5.3. You can get it from ftp.ee.lbl.gov. 5467 5468 Vern 5469 5470 5471File: flex.info, Node: The | operator is not doing what I want, Next: Why can't flex understand this variable trailing context pattern?, Prev: unput() messes up yy_at_bol, Up: FAQ 5472 5473The | operator is not doing what I want 5474======================================= 5475 5476 To: Alain.ISSARD@st.com 5477 Subject: Re: Start condition with FLEX 5478 In-reply-to: Your message of Mon, 18 Nov 1996 09:45:02 PST. 5479 Date: Mon, 18 Nov 1996 10:41:34 PST 5480 From: Vern Paxson <vern> 5481 5482 > I am not able to use the start condition scope and to use the | (OR) with 5483 > rules having start conditions. 5484 5485 The problem is that if you use '|' as a regular expression operator, for 5486 example "a|b" meaning "match either 'a' or 'b'", then it must *not* have 5487 any blanks around it. If you instead want the special '|' *action* (which 5488 from your scanner appears to be the case), which is a way of giving two 5489 different rules the same action: 5490 5491 foo | 5492 bar matched_foo_or_bar(); 5493 5494 then '|' *must* be separated from the first rule by whitespace and *must* 5495 be followed by a new line. You *cannot* write it as: 5496 5497 foo | bar matched_foo_or_bar(); 5498 5499 even though you might think you could because yacc supports this syntax. 5500 The reason for this unfortunately incompatibility is historical, but it's 5501 unlikely to be changed. 5502 5503 Your problems with start condition scope are simply due to syntax errors 5504 from your use of '|' later confusing flex. 5505 5506 Let me know if you still have problems. 5507 5508 Vern 5509 5510 5511File: flex.info, Node: Why can't flex understand this variable trailing context pattern?, Next: The ^ operator isn't working, Prev: The | operator is not doing what I want, Up: FAQ 5512 5513Why can't flex understand this variable trailing context pattern? 5514================================================================= 5515 5516 To: Gregory Margo <gmargo@newton.vip.best.com> 5517 Subject: Re: flex-2.5.3 bug report 5518 In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST. 5519 Date: Sat, 23 Nov 1996 17:07:32 PST 5520 From: Vern Paxson <vern> 5521 5522 > Enclosed is a lex file that "real" lex will process, but I cannot get 5523 > flex to process it. Could you try it and maybe point me in the right direction? 5524 5525 Your problem is that some of the definitions in the scanner use the '/' 5526 trailing context operator, and have it enclosed in ()'s. Flex does not 5527 allow this operator to be enclosed in ()'s because doing so allows undefined 5528 regular expressions such as "(a/b)+". So the solution is to remove the 5529 parentheses. Note that you must also be building the scanner with the -l 5530 option for AT&T lex compatibility. Without this option, flex automatically 5531 encloses the definitions in parentheses. 5532 5533 Vern 5534 5535 5536File: flex.info, Node: The ^ operator isn't working, Next: Trailing context is getting confused with trailing optional patterns, Prev: Why can't flex understand this variable trailing context pattern?, Up: FAQ 5537 5538The ^ operator isn't working 5539============================ 5540 5541 To: Thomas Hadig <hadig@toots.physik.rwth-aachen.de> 5542 Subject: Re: Flex Bug ? 5543 In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST. 5544 Date: Tue, 26 Nov 1996 11:15:05 PST 5545 From: Vern Paxson <vern> 5546 5547 > In my lexer code, i have the line : 5548 > ^\*.* { } 5549 > 5550 > Thus all lines starting with an astrix (*) are comment lines. 5551 > This does not work ! 5552 5553 I can't get this problem to reproduce - it works fine for me. Note 5554 though that if what you have is slightly different: 5555 5556 COMMENT ^\*.* 5557 %% 5558 {COMMENT} { } 5559 5560 then it won't work, because flex pushes back macro definitions enclosed 5561 in ()'s, so the rule becomes 5562 5563 (^\*.*) { } 5564 5565 and now that the '^' operator is not at the immediate beginning of the 5566 line, it's interpreted as just a regular character. You can avoid this 5567 behavior by using the "-l" lex-compatibility flag, or "%option lex-compat". 5568 5569 Vern 5570 5571 5572File: flex.info, Node: Trailing context is getting confused with trailing optional patterns, Next: Is flex GNU or not?, Prev: The ^ operator isn't working, Up: FAQ 5573 5574Trailing context is getting confused with trailing optional patterns 5575==================================================================== 5576 5577 To: Adoram Rogel <adoram@hybridge.com> 5578 Subject: Re: Flex 2.5.4 BOF ??? 5579 In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST. 5580 Date: Wed, 27 Nov 1996 10:56:25 PST 5581 From: Vern Paxson <vern> 5582 5583 > Organization(s)?/[a-z] 5584 > 5585 > This matched "Organizations" (looking in debug mode, the trailing s 5586 > was matched with trailing context instead of the optional (s) in the 5587 > end of the word. 5588 5589 That should only happen with lex. Flex can properly match this pattern. 5590 (That might be what you're saying, I'm just not sure.) 5591 5592 > Is there a way to avoid this dangerous trailing context problem ? 5593 5594 Unfortunately, there's no easy way. On the other hand, I don't see why 5595 it should be a problem. Lex's matching is clearly wrong, and I'd hope 5596 that usually the intent remains the same as expressed with the pattern, 5597 so flex's matching will be correct. 5598 5599 Vern 5600 5601 5602File: flex.info, Node: Is flex GNU or not?, Next: ERASEME53, Prev: Trailing context is getting confused with trailing optional patterns, Up: FAQ 5603 5604Is flex GNU or not? 5605=================== 5606 5607 To: Cameron MacKinnon <mackin@interlog.com> 5608 Subject: Re: Flex documentation bug 5609 In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST. 5610 Date: Sun, 01 Dec 1996 22:29:39 PST 5611 From: Vern Paxson <vern> 5612 5613 > I'm not sure how or where to submit bug reports (documentation or 5614 > otherwise) for the GNU project stuff ... 5615 5616 Well, strictly speaking flex isn't part of the GNU project. They just 5617 distribute it because no one's written a decent GPL'd lex replacement. 5618 So you should send bugs directly to me. Those sent to the GNU folks 5619 sometimes find there way to me, but some may drop between the cracks. 5620 5621 > In GNU Info, under the section 'Start Conditions', and also in the man 5622 > page (mine's dated April '95) is a nice little snippet showing how to 5623 > parse C quoted strings into a buffer, defined to be MAX_STR_CONST in 5624 > size. Unfortunately, no overflow checking is ever done ... 5625 5626 This is already mentioned in the manual: 5627 5628 Finally, here's an example of how to match C-style quoted 5629 strings using exclusive start conditions, including expanded 5630 escape sequences (but not including checking for a string 5631 that's too long): 5632 5633 The reason for not doing the overflow checking is that it will needlessly 5634 clutter up an example whose main purpose is just to demonstrate how to 5635 use flex. 5636 5637 The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov. 5638 5639 Vern 5640 5641 5642File: flex.info, Node: ERASEME53, Next: I need to scan if-then-else blocks and while loops, Prev: Is flex GNU or not?, Up: FAQ 5643 5644ERASEME53 5645========= 5646 5647 To: tsv@cs.UManitoba.CA 5648 Subject: Re: Flex (reg).. 5649 In-reply-to: Your message of Thu, 06 Mar 1997 23:50:16 PST. 5650 Date: Thu, 06 Mar 1997 15:54:19 PST 5651 From: Vern Paxson <vern> 5652 5653 > [:alpha:] ([:alnum:] | \\_)* 5654 5655 If your rule really has embedded blanks as shown above, then it won't 5656 work, as the first blank delimits the rule from the action. (It wouldn't 5657 even compile ...) You need instead: 5658 5659 [:alpha:]([:alnum:]|\\_)* 5660 5661 and that should work fine - there's no restriction on what can go inside 5662 of ()'s except for the trailing context operator, '/'. 5663 5664 Vern 5665 5666 5667File: flex.info, Node: I need to scan if-then-else blocks and while loops, Next: ERASEME55, Prev: ERASEME53, Up: FAQ 5668 5669I need to scan if-then-else blocks and while loops 5670================================================== 5671 5672 To: "Mike Stolnicki" <mstolnic@ford.com> 5673 Subject: Re: FLEX help 5674 In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT. 5675 Date: Fri, 30 May 1997 10:46:35 PDT 5676 From: Vern Paxson <vern> 5677 5678 > We'd like to add "if-then-else", "while", and "for" statements to our 5679 > language ... 5680 > We've investigated many possible solutions. The one solution that seems 5681 > the most reasonable involves knowing the position of a TOKEN in yyin. 5682 5683 I strongly advise you to instead build a parse tree (abstract syntax tree) 5684 and loop over that instead. You'll find this has major benefits in keeping 5685 your interpreter simple and extensible. 5686 5687 That said, the functionality you mention for get_position and set_position 5688 have been on the to-do list for a while. As flex is a purely spare-time 5689 project for me, no guarantees when this will be added (in particular, it 5690 for sure won't be for many months to come). 5691 5692 Vern 5693 5694 5695File: flex.info, Node: ERASEME55, Next: ERASEME56, Prev: I need to scan if-then-else blocks and while loops, Up: FAQ 5696 5697ERASEME55 5698========= 5699 5700 To: Colin Paul Adams <colin@colina.demon.co.uk> 5701 Subject: Re: Flex C++ classes and Bison 5702 In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT. 5703 Date: Fri, 15 Aug 1997 10:48:19 PDT 5704 From: Vern Paxson <vern> 5705 5706 > #define YY_DECL int yylex (YYSTYPE *lvalp, struct parser_control 5707 > *parm) 5708 > 5709 > I have been trying to get this to work as a C++ scanner, but it does 5710 > not appear to be possible (warning that it matches no declarations in 5711 > yyFlexLexer, or something like that). 5712 > 5713 > Is this supposed to be possible, or is it being worked on (I DID 5714 > notice the comment that scanner classes are still experimental, so I'm 5715 > not too hopeful)? 5716 5717 What you need to do is derive a subclass from yyFlexLexer that provides 5718 the above yylex() method, squirrels away lvalp and parm into member 5719 variables, and then invokes yyFlexLexer::yylex() to do the regular scanning. 5720 5721 Vern 5722 5723 5724File: flex.info, Node: ERASEME56, Next: ERASEME57, Prev: ERASEME55, Up: FAQ 5725 5726ERASEME56 5727========= 5728 5729 To: Mikael.Latvala@lmf.ericsson.se 5730 Subject: Re: Possible mistake in Flex v2.5 document 5731 In-reply-to: Your message of Fri, 05 Sep 1997 16:07:24 PDT. 5732 Date: Fri, 05 Sep 1997 10:01:54 PDT 5733 From: Vern Paxson <vern> 5734 5735 > In that example you show how to count comment lines when using 5736 > C style /* ... */ comments. My question is, shouldn't you take into 5737 > account a scenario where end of a comment marker occurs inside 5738 > character or string literals? 5739 5740 The scanner certainly needs to also scan character and string literals. 5741 However it does that (there's an example in the man page for strings), the 5742 lexer will recognize the beginning of the literal before it runs across the 5743 embedded "/*". Consequently, it will finish scanning the literal before it 5744 even considers the possibility of matching "/*". 5745 5746 Example: 5747 5748 '([^']*|{ESCAPE_SEQUENCE})' 5749 5750 will match all the text between the ''s (inclusive). So the lexer 5751 considers this as a token beginning at the first ', and doesn't even 5752 attempt to match other tokens inside it. 5753 5754 I thinnk this subtlety is not worth putting in the manual, as I suspect 5755 it would confuse more people than it would enlighten. 5756 5757 Vern 5758 5759 5760File: flex.info, Node: ERASEME57, Next: Is there a repository for flex scanners?, Prev: ERASEME56, Up: FAQ 5761 5762ERASEME57 5763========= 5764 5765 To: "Marty Leisner" <leisner@sdsp.mc.xerox.com> 5766 Subject: Re: flex limitations 5767 In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT. 5768 Date: Mon, 08 Sep 1997 11:38:08 PDT 5769 From: Vern Paxson <vern> 5770 5771 > %% 5772 > [a-zA-Z]+ /* skip a line */ 5773 > { printf("got %s\n", yytext); } 5774 > %% 5775 5776 What version of flex are you using? If I feed this to 2.5.4, it complains: 5777 5778 "bug.l", line 5: EOF encountered inside an action 5779 "bug.l", line 5: unrecognized rule 5780 "bug.l", line 5: fatal parse error 5781 5782 Not the world's greatest error message, but it manages to flag the problem. 5783 5784 (With the introduction of start condition scopes, flex can't accommodate 5785 an action on a separate line, since it's ambiguous with an indented rule.) 5786 5787 You can get 2.5.4 from ftp.ee.lbl.gov. 5788 5789 Vern 5790 5791 5792File: flex.info, Node: Is there a repository for flex scanners?, Next: How can I conditionally compile or preprocess my flex input file?, Prev: ERASEME57, Up: FAQ 5793 5794Is there a repository for flex scanners? 5795======================================== 5796 5797Not that we know of. You might try asking on comp.compilers. 5798 5799 5800File: flex.info, Node: How can I conditionally compile or preprocess my flex input file?, Next: Where can I find grammars for lex and yacc?, Prev: Is there a repository for flex scanners?, Up: FAQ 5801 5802How can I conditionally compile or preprocess my flex input file? 5803================================================================= 5804 5805Flex doesn't have a preprocessor like C does. You might try using m4, 5806or the C preprocessor plus a sed script to clean up the result. 5807 5808 5809File: flex.info, Node: Where can I find grammars for lex and yacc?, Next: I get an end-of-buffer message for each character scanned., Prev: How can I conditionally compile or preprocess my flex input file?, Up: FAQ 5810 5811Where can I find grammars for lex and yacc? 5812=========================================== 5813 5814In the sources for flex and bison. 5815 5816 5817File: flex.info, Node: I get an end-of-buffer message for each character scanned., Next: unnamed-faq-62, Prev: Where can I find grammars for lex and yacc?, Up: FAQ 5818 5819I get an end-of-buffer message for each character scanned. 5820========================================================== 5821 5822This will happen if your LexerInput() function returns only one 5823character at a time, which can happen either if you're scanner is 5824"interactive", or if the streams library on your platform always returns 58251 for yyin->gcount(). 5826 5827 Solution: override LexerInput() with a version that returns whole 5828buffers. 5829 5830 5831File: flex.info, Node: unnamed-faq-62, Next: unnamed-faq-63, Prev: I get an end-of-buffer message for each character scanned., Up: FAQ 5832 5833unnamed-faq-62 5834============== 5835 5836 To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 5837 Subject: Re: Flex maximums 5838 In-reply-to: Your message of Mon, 17 Nov 1997 17:16:06 PST. 5839 Date: Mon, 17 Nov 1997 17:16:15 PST 5840 From: Vern Paxson <vern> 5841 5842 > I took a quick look into the flex-sources and altered some #defines in 5843 > flexdefs.h: 5844 > 5845 > #define INITIAL_MNS 64000 5846 > #define MNS_INCREMENT 1024000 5847 > #define MAXIMUM_MNS 64000 5848 5849 The things to fix are to add a couple of zeroes to: 5850 5851 #define JAMSTATE -32766 /* marks a reference to the state that always jams */ 5852 #define MAXIMUM_MNS 31999 5853 #define BAD_SUBSCRIPT -32767 5854 #define MAX_SHORT 32700 5855 5856 and, if you get complaints about too many rules, make the following change too: 5857 5858 #define YY_TRAILING_MASK 0x200000 5859 #define YY_TRAILING_HEAD_MASK 0x400000 5860 5861 - Vern 5862 5863 5864File: flex.info, Node: unnamed-faq-63, Next: unnamed-faq-64, Prev: unnamed-faq-62, Up: FAQ 5865 5866unnamed-faq-63 5867============== 5868 5869 To: jimmey@lexis-nexis.com (Jimmey Todd) 5870 Subject: Re: FLEX question regarding istream vs ifstream 5871 In-reply-to: Your message of Mon, 08 Dec 1997 15:54:15 PST. 5872 Date: Mon, 15 Dec 1997 13:21:35 PST 5873 From: Vern Paxson <vern> 5874 5875 > stdin_handle = YY_CURRENT_BUFFER; 5876 > ifstream fin( "aFile" ); 5877 > yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) ); 5878 > 5879 > What I'm wanting to do, is pass the contents of a file thru one set 5880 > of rules and then pass stdin thru another set... It works great if, I 5881 > don't use the C++ classes. But since everything else that I'm doing is 5882 > in C++, I thought I'd be consistent. 5883 > 5884 > The problem is that 'yy_create_buffer' is expecting an istream* as it's 5885 > first argument (as stated in the man page). However, fin is a ifstream 5886 > object. Any ideas on what I might be doing wrong? Any help would be 5887 > appreciated. Thanks!! 5888 5889 You need to pass &fin, to turn it into an ifstream* instead of an ifstream. 5890 Then its type will be compatible with the expected istream*, because ifstream 5891 is derived from istream. 5892 5893 Vern 5894 5895 5896File: flex.info, Node: unnamed-faq-64, Next: unnamed-faq-65, Prev: unnamed-faq-63, Up: FAQ 5897 5898unnamed-faq-64 5899============== 5900 5901 To: Enda Fadian <fadiane@piercom.ie> 5902 Subject: Re: Question related to Flex man page? 5903 In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST. 5904 Date: Tue, 16 Dec 1997 14:17:09 PST 5905 From: Vern Paxson <vern> 5906 5907 > Can you explain to me what is ment by a long-jump in relation to flex? 5908 5909 Using the longjmp() function while inside yylex() or a routine called by it. 5910 5911 > what is the flex activation frame. 5912 5913 Just yylex()'s stack frame. 5914 5915 > As far as I can see yyrestart will bring me back to the sart of the input 5916 > file and using flex++ isnot really an option! 5917 5918 No, yyrestart() doesn't imply a rewind, even though its name might sound 5919 like it does. It tells the scanner to flush its internal buffers and 5920 start reading from the given file at its present location. 5921 5922 Vern 5923 5924 5925File: flex.info, Node: unnamed-faq-65, Next: unnamed-faq-66, Prev: unnamed-faq-64, Up: FAQ 5926 5927unnamed-faq-65 5928============== 5929 5930 To: hassan@larc.info.uqam.ca (Hassan Alaoui) 5931 Subject: Re: Need urgent Help 5932 In-reply-to: Your message of Sat, 20 Dec 1997 19:38:19 PST. 5933 Date: Sun, 21 Dec 1997 21:30:46 PST 5934 From: Vern Paxson <vern> 5935 5936 > /usr/lib/yaccpar: In function `int yyparse()': 5937 > /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)' 5938 > 5939 > ld: Undefined symbol 5940 > _yylex 5941 > _yyparse 5942 > _yyin 5943 5944 This is a known problem with Solaris C++ (and/or Solaris yacc). I believe 5945 the fix is to explicitly insert some 'extern "C"' statements for the 5946 corresponding routines/symbols. 5947 5948 Vern 5949 5950 5951File: flex.info, Node: unnamed-faq-66, Next: unnamed-faq-67, Prev: unnamed-faq-65, Up: FAQ 5952 5953unnamed-faq-66 5954============== 5955 5956 To: mc0307@mclink.it 5957 Cc: gnu@prep.ai.mit.edu 5958 Subject: Re: [mc0307@mclink.it: Help request] 5959 In-reply-to: Your message of Fri, 12 Dec 1997 17:57:29 PST. 5960 Date: Sun, 21 Dec 1997 22:33:37 PST 5961 From: Vern Paxson <vern> 5962 5963 > This is my definition for float and integer types: 5964 > . . . 5965 > NZD [1-9] 5966 > ... 5967 > I've tested my program on other lex version (on UNIX Sun Solaris an HP 5968 > UNIX) and it work well, so I think that my definitions are correct. 5969 > There are any differences between Lex and Flex? 5970 5971 There are indeed differences, as discussed in the man page. The one 5972 you are probably running into is that when flex expands a name definition, 5973 it puts parentheses around the expansion, while lex does not. There's 5974 an example in the man page of how this can lead to different matching. 5975 Flex's behavior complies with the POSIX standard (or at least with the 5976 last POSIX draft I saw). 5977 5978 Vern 5979 5980 5981File: flex.info, Node: unnamed-faq-67, Next: unnamed-faq-68, Prev: unnamed-faq-66, Up: FAQ 5982 5983unnamed-faq-67 5984============== 5985 5986 To: hassan@larc.info.uqam.ca (Hassan Alaoui) 5987 Subject: Re: Thanks 5988 In-reply-to: Your message of Mon, 22 Dec 1997 16:06:35 PST. 5989 Date: Mon, 22 Dec 1997 14:35:05 PST 5990 From: Vern Paxson <vern> 5991 5992 > Thank you very much for your help. I compile and link well with C++ while 5993 > declaring 'yylex ...' extern, But a little problem remains. I get a 5994 > segmentation default when executing ( I linked with lfl library) while it 5995 > works well when using LEX instead of flex. Do you have some ideas about the 5996 > reason for this ? 5997 5998 The one possible reason for this that comes to mind is if you've defined 5999 yytext as "extern char yytext[]" (which is what lex uses) instead of 6000 "extern char *yytext" (which is what flex uses). If it's not that, then 6001 I'm afraid I don't know what the problem might be. 6002 6003 Vern 6004 6005 6006File: flex.info, Node: unnamed-faq-68, Next: unnamed-faq-69, Prev: unnamed-faq-67, Up: FAQ 6007 6008unnamed-faq-68 6009============== 6010 6011 To: "Bart Niswonger" <NISWONGR@almaden.ibm.com> 6012 Subject: Re: flex 2.5: c++ scanners & start conditions 6013 In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST. 6014 Date: Tue, 06 Jan 1998 19:19:30 PST 6015 From: Vern Paxson <vern> 6016 6017 > The problem is that when I do this (using %option c++) start 6018 > conditions seem to not apply. 6019 6020 The BEGIN macro modifies the yy_start variable. For C scanners, this 6021 is a static with scope visible through the whole file. For C++ scanners, 6022 it's a member variable, so it only has visible scope within a member 6023 function. Your lexbegin() routine is not a member function when you 6024 build a C++ scanner, so it's not modifying the correct yy_start. The 6025 diagnostic that indicates this is that you found you needed to add 6026 a declaration of yy_start in order to get your scanner to compile when 6027 using C++; instead, the correct fix is to make lexbegin() a member 6028 function (by deriving from yyFlexLexer). 6029 6030 Vern 6031 6032 6033File: flex.info, Node: unnamed-faq-69, Next: unnamed-faq-70, Prev: unnamed-faq-68, Up: FAQ 6034 6035unnamed-faq-69 6036============== 6037 6038 To: "Boris Zinin" <boris@ippe.rssi.ru> 6039 Subject: Re: current position in flex buffer 6040 In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST. 6041 Date: Mon, 12 Jan 1998 12:03:15 PST 6042 From: Vern Paxson <vern> 6043 6044 > The problem is how to determine the current position in flex active 6045 > buffer when a rule is matched.... 6046 6047 You will need to keep track of this explicitly, such as by redefining 6048 YY_USER_ACTION to count the number of characters matched. 6049 6050 The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov. 6051 6052 Vern 6053 6054 6055File: flex.info, Node: unnamed-faq-70, Next: unnamed-faq-71, Prev: unnamed-faq-69, Up: FAQ 6056 6057unnamed-faq-70 6058============== 6059 6060 To: Bik.Dhaliwal@bis.org 6061 Subject: Re: Flex question 6062 In-reply-to: Your message of Mon, 26 Jan 1998 13:05:35 PST. 6063 Date: Tue, 27 Jan 1998 22:41:52 PST 6064 From: Vern Paxson <vern> 6065 6066 > That requirement involves knowing 6067 > the character position at which a particular token was matched 6068 > in the lexer. 6069 6070 The way you have to do this is by explicitly keeping track of where 6071 you are in the file, by counting the number of characters scanned 6072 for each token (available in yyleng). It may prove convenient to 6073 do this by redefining YY_USER_ACTION, as described in the manual. 6074 6075 Vern 6076 6077 6078File: flex.info, Node: unnamed-faq-71, Next: unnamed-faq-72, Prev: unnamed-faq-70, Up: FAQ 6079 6080unnamed-faq-71 6081============== 6082 6083 To: Vladimir Alexiev <vladimir@cs.ualberta.ca> 6084 Subject: Re: flex: how to control start condition from parser? 6085 In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST. 6086 Date: Tue, 27 Jan 1998 22:45:37 PST 6087 From: Vern Paxson <vern> 6088 6089 > It seems useful for the parser to be able to tell the lexer about such 6090 > context dependencies, because then they don't have to be limited to 6091 > local or sequential context. 6092 6093 One way to do this is to have the parser call a stub routine that's 6094 included in the scanner's .l file, and consequently that has access ot 6095 BEGIN. The only ugliness is that the parser can't pass in the state 6096 it wants, because those aren't visible - but if you don't have many 6097 such states, then using a different set of names doesn't seem like 6098 to much of a burden. 6099 6100 While generating a .h file like you suggests is certainly cleaner, 6101 flex development has come to a virtual stand-still :-(, so a workaround 6102 like the above is much more pragmatic than waiting for a new feature. 6103 6104 Vern 6105 6106 6107File: flex.info, Node: unnamed-faq-72, Next: unnamed-faq-73, Prev: unnamed-faq-71, Up: FAQ 6108 6109unnamed-faq-72 6110============== 6111 6112 To: Barbara Denny <denny@3com.com> 6113 Subject: Re: freebsd flex bug? 6114 In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST. 6115 Date: Fri, 30 Jan 1998 12:42:32 PST 6116 From: Vern Paxson <vern> 6117 6118 > lex.yy.c:1996: parse error before `=' 6119 6120 This is the key, identifying this error. (It may help to pinpoint 6121 it by using flex -L, so it doesn't generate #line directives in its 6122 output.) I will bet you heavy money that you have a start condition 6123 name that is also a variable name, or something like that; flex spits 6124 out #define's for each start condition name, mapping them to a number, 6125 so you can wind up with: 6126 6127 %x foo 6128 %% 6129 ... 6130 %% 6131 void bar() 6132 { 6133 int foo = 3; 6134 } 6135 6136 and the penultimate will turn into "int 1 = 3" after C preprocessing, 6137 since flex will put "#define foo 1" in the generated scanner. 6138 6139 Vern 6140 6141 6142File: flex.info, Node: unnamed-faq-73, Next: unnamed-faq-74, Prev: unnamed-faq-72, Up: FAQ 6143 6144unnamed-faq-73 6145============== 6146 6147 To: Maurice Petrie <mpetrie@infoscigroup.com> 6148 Subject: Re: Lost flex .l file 6149 In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST. 6150 Date: Mon, 02 Feb 1998 11:15:12 PST 6151 From: Vern Paxson <vern> 6152 6153 > I am curious as to 6154 > whether there is a simple way to backtrack from the generated source to 6155 > reproduce the lost list of tokens we are searching on. 6156 6157 In theory, it's straight-forward to go from the DFA representation 6158 back to a regular-expression representation - the two are isomorphic. 6159 In practice, a huge headache, because you have to unpack all the tables 6160 back into a single DFA representation, and then write a program to munch 6161 on that and translate it into an RE. 6162 6163 Sorry for the less-than-happy news ... 6164 6165 Vern 6166 6167 6168File: flex.info, Node: unnamed-faq-74, Next: unnamed-faq-75, Prev: unnamed-faq-73, Up: FAQ 6169 6170unnamed-faq-74 6171============== 6172 6173 To: jimmey@lexis-nexis.com (Jimmey Todd) 6174 Subject: Re: Flex performance question 6175 In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. 6176 Date: Thu, 19 Feb 1998 08:48:51 PST 6177 From: Vern Paxson <vern> 6178 6179 > What I have found, is that the smaller the data chunk, the faster the 6180 > program executes. This is the opposite of what I expected. Should this be 6181 > happening this way? 6182 6183 This is exactly what will happen if your input file has embedded NULs. 6184 From the man page: 6185 6186 A final note: flex is slow when matching NUL's, particularly 6187 when a token contains multiple NUL's. It's best to write 6188 rules which match short amounts of text if it's anticipated 6189 that the text will often include NUL's. 6190 6191 So that's the first thing to look for. 6192 6193 Vern 6194 6195 6196File: flex.info, Node: unnamed-faq-75, Next: unnamed-faq-76, Prev: unnamed-faq-74, Up: FAQ 6197 6198unnamed-faq-75 6199============== 6200 6201 To: jimmey@lexis-nexis.com (Jimmey Todd) 6202 Subject: Re: Flex performance question 6203 In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. 6204 Date: Thu, 19 Feb 1998 15:42:25 PST 6205 From: Vern Paxson <vern> 6206 6207 So there are several problems. 6208 6209 First, to go fast, you want to match as much text as possible, which 6210 your scanners don't in the case that what they're scanning is *not* 6211 a <RN> tag. So you want a rule like: 6212 6213 [^<]+ 6214 6215 Second, C++ scanners are particularly slow if they're interactive, 6216 which they are by default. Using -B speeds it up by a factor of 3-4 6217 on my workstation. 6218 6219 Third, C++ scanners that use the istream interface are slow, because 6220 of how poorly implemented istream's are. I built two versions of 6221 the following scanner: 6222 6223 %% 6224 .*\n 6225 .* 6226 %% 6227 6228 and the C version inhales a 2.5MB file on my workstation in 0.8 seconds. 6229 The C++ istream version, using -B, takes 3.8 seconds. 6230 6231 Vern 6232 6233 6234File: flex.info, Node: unnamed-faq-76, Next: unnamed-faq-77, Prev: unnamed-faq-75, Up: FAQ 6235 6236unnamed-faq-76 6237============== 6238 6239 To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com> 6240 Subject: Re: FLEX 2.5 & THE YEAR 2000 6241 In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT. 6242 Date: Wed, 03 Jun 1998 10:22:26 PDT 6243 From: Vern Paxson <vern> 6244 6245 > I am researching the Y2K problem with General Electric R&D 6246 > and need to know if there are any known issues concerning 6247 > the above mentioned software and Y2K regardless of version. 6248 6249 There shouldn't be, all it ever does with the date is ask the system 6250 for it and then print it out. 6251 6252 Vern 6253 6254 6255File: flex.info, Node: unnamed-faq-77, Next: unnamed-faq-78, Prev: unnamed-faq-76, Up: FAQ 6256 6257unnamed-faq-77 6258============== 6259 6260 To: "Hans Dermot Doran" <htd@ibhdoran.com> 6261 Subject: Re: flex problem 6262 In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT. 6263 Date: Tue, 21 Jul 1998 14:23:34 PDT 6264 From: Vern Paxson <vern> 6265 6266 > To overcome this, I gets() the stdin into a string and lex the string. The 6267 > string is lexed OK except that the end of string isn't lexed properly 6268 > (yy_scan_string()), that is the lexer dosn't recognise the end of string. 6269 6270 Flex doesn't contain mechanisms for recognizing buffer endpoints. But if 6271 you use fgets instead (which you should anyway, to protect against buffer 6272 overflows), then the final \n will be preserved in the string, and you can 6273 scan that in order to find the end of the string. 6274 6275 Vern 6276 6277 6278File: flex.info, Node: unnamed-faq-78, Next: unnamed-faq-79, Prev: unnamed-faq-77, Up: FAQ 6279 6280unnamed-faq-78 6281============== 6282 6283 To: soumen@almaden.ibm.com 6284 Subject: Re: Flex++ 2.5.3 instance member vs. static member 6285 In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT. 6286 Date: Tue, 28 Jul 1998 01:10:34 PDT 6287 From: Vern Paxson <vern> 6288 6289 > %{ 6290 > int mylineno = 0; 6291 > %} 6292 > ws [ \t]+ 6293 > alpha [A-Za-z] 6294 > dig [0-9] 6295 > %% 6296 > 6297 > Now you'd expect mylineno to be a member of each instance of class 6298 > yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to 6299 > indicate otherwise; unless I am missing something the declaration of 6300 > mylineno seems to be outside any class scope. 6301 > 6302 > How will this work if I want to run a multi-threaded application with each 6303 > thread creating a FlexLexer instance? 6304 6305 Derive your own subclass and make mylineno a member variable of it. 6306 6307 Vern 6308 6309 6310File: flex.info, Node: unnamed-faq-79, Next: unnamed-faq-80, Prev: unnamed-faq-78, Up: FAQ 6311 6312unnamed-faq-79 6313============== 6314 6315 To: Adoram Rogel <adoram@hybridge.com> 6316 Subject: Re: More than 32K states change hangs 6317 In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT. 6318 Date: Tue, 04 Aug 1998 22:28:45 PDT 6319 From: Vern Paxson <vern> 6320 6321 > Vern Paxson, 6322 > 6323 > I followed your advice, posted on Usenet bu you, and emailed to me 6324 > personally by you, on how to overcome the 32K states limit. I'm running 6325 > on Linux machines. 6326 > I took the full source of version 2.5.4 and did the following changes in 6327 > flexdef.h: 6328 > #define JAMSTATE -327660 6329 > #define MAXIMUM_MNS 319990 6330 > #define BAD_SUBSCRIPT -327670 6331 > #define MAX_SHORT 327000 6332 > 6333 > and compiled. 6334 > All looked fine, including check and bigcheck, so I installed. 6335 6336 Hmmm, you shouldn't increase MAX_SHORT, though looking through my email 6337 archives I see that I did indeed recommend doing so. Try setting it back 6338 to 32700; that should suffice that you no longer need -Ca. If it still 6339 hangs, then the interesting question is - where? 6340 6341 > Compiling the same hanged program with a out-of-the-box (RedHat 4.2 6342 > distribution of Linux) 6343 > flex 2.5.4 binary works. 6344 6345 Since Linux comes with source code, you should diff it against what 6346 you have to see what problems they missed. 6347 6348 > Should I always compile with the -Ca option now ? even short and simple 6349 > filters ? 6350 6351 No, definitely not. It's meant to be for those situations where you 6352 absolutely must squeeze every last cycle out of your scanner. 6353 6354 Vern 6355 6356 6357File: flex.info, Node: unnamed-faq-80, Next: unnamed-faq-81, Prev: unnamed-faq-79, Up: FAQ 6358 6359unnamed-faq-80 6360============== 6361 6362 To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com> 6363 Subject: Re: flex output for static code portion 6364 In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT. 6365 Date: Mon, 17 Aug 1998 23:57:42 PDT 6366 From: Vern Paxson <vern> 6367 6368 > I would like to use flex under the hood to generate a binary file 6369 > containing the data structures that control the parse. 6370 6371 This has been on the wish-list for a long time. In principle it's 6372 straight-forward - you redirect mkdata() et al's I/O to another file, 6373 and modify the skeleton to have a start-up function that slurps these 6374 into dynamic arrays. The concerns are (1) the scanner generation code 6375 is hairy and full of corner cases, so it's easy to get surprised when 6376 going down this path :-( ; and (2) being careful about buffering so 6377 that when the tables change you make sure the scanner starts in the 6378 correct state and reading at the right point in the input file. 6379 6380 > I was wondering if you know of anyone who has used flex in this way. 6381 6382 I don't - but it seems like a reasonable project to undertake (unlike 6383 numerous other flex tweaks :-). 6384 6385 Vern 6386 6387 6388File: flex.info, Node: unnamed-faq-81, Next: unnamed-faq-82, Prev: unnamed-faq-80, Up: FAQ 6389 6390unnamed-faq-81 6391============== 6392 6393 Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11]) 6394 by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838 6395 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT) 6396 Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2]) 6397 by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694 6398 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200 6399 Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200 6400 From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de> 6401 Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de> 6402 Subject: "flex scanner push-back overflow" 6403 To: vern@ee.lbl.gov 6404 Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST) 6405 Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 6406 X-NoJunk: Do NOT send commercial mail, spam or ads to this address! 6407 X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/ 6408 X-Mailer: ELM [version 2.4ME+ PL28 (25)] 6409 MIME-Version: 1.0 6410 Content-Type: text/plain; charset=US-ASCII 6411 Content-Transfer-Encoding: 7bit 6412 6413 Hi Vern, 6414 6415 Yesterday, I encountered a strange problem: I use the macro processor m4 6416 to include some lengthy lists into a .l file. Following is a flex macro 6417 definition that causes some serious pain in my neck: 6418 6419 AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...]) 6420 6421 The complete list contains about 10kB. When I try to "flex" this file 6422 (on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased 6423 some of the predefined values in flexdefs.h) I get the error: 6424 6425 myflex/flex -8 sentag.tmp.l 6426 flex scanner push-back overflow 6427 6428 When I remove the slashes in the macro definition everything works fine. 6429 As I understand it, the double quotes escape the slash-character so it 6430 really means "/" and not "trailing context". Furthermore, I tried to 6431 escape the slashes with backslashes, but with no use, the same error message 6432 appeared when flexing the code. 6433 6434 Do you have an idea what's going on here? 6435 6436 Greetings from Germany, 6437 Georg 6438 -- 6439 Georg Rehm georg@cl-ki.uni-osnabrueck.de 6440 Institute for Semantic Information Processing, University of Osnabrueck, FRG 6441 6442 6443File: flex.info, Node: unnamed-faq-82, Next: unnamed-faq-83, Prev: unnamed-faq-81, Up: FAQ 6444 6445unnamed-faq-82 6446============== 6447 6448 To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE 6449 Subject: Re: "flex scanner push-back overflow" 6450 In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT. 6451 Date: Thu, 20 Aug 1998 07:05:35 PDT 6452 From: Vern Paxson <vern> 6453 6454 > myflex/flex -8 sentag.tmp.l 6455 > flex scanner push-back overflow 6456 6457 Flex itself uses a flex scanner. That scanner is running out of buffer 6458 space when it tries to unput() the humongous macro you've defined. When 6459 you remove the '/'s, you make it small enough so that it fits in the buffer; 6460 removing spaces would do the same thing. 6461 6462 The fix is to either rethink how come you're using such a big macro and 6463 perhaps there's another/better way to do it; or to rebuild flex's own 6464 scan.c with a larger value for 6465 6466 #define YY_BUF_SIZE 16384 6467 6468 - Vern 6469 6470 6471File: flex.info, Node: unnamed-faq-83, Next: unnamed-faq-84, Prev: unnamed-faq-82, Up: FAQ 6472 6473unnamed-faq-83 6474============== 6475 6476 To: Jan Kort <jan@research.techforce.nl> 6477 Subject: Re: Flex 6478 In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200. 6479 Date: Sat, 05 Sep 1998 00:59:49 PDT 6480 From: Vern Paxson <vern> 6481 6482 > %% 6483 > 6484 > "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); } 6485 > ^\n { fprintf(stderr, "empty line\n"); } 6486 > . { } 6487 > \n { fprintf(stderr, "new line\n"); } 6488 > 6489 > %% 6490 > -- input --------------------------------------- 6491 > TEST1 6492 > -- output -------------------------------------- 6493 > TEST1 6494 > empty line 6495 > ------------------------------------------------ 6496 6497 IMHO, it's not clear whether or not this is in fact a bug. It depends 6498 on whether you view yyless() as backing up in the input stream, or as 6499 pushing new characters onto the beginning of the input stream. Flex 6500 interprets it as the latter (for implementation convenience, I'll admit), 6501 and so considers the newline as in fact matching at the beginning of a 6502 line, as after all the last token scanned an entire line and so the 6503 scanner is now at the beginning of a new line. 6504 6505 I agree that this is counter-intuitive for yyless(), given its 6506 functional description (it's less so for unput(), depending on whether 6507 you're unput()'ing new text or scanned text). But I don't plan to 6508 change it any time soon, as it's a pain to do so. Consequently, 6509 you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak 6510 your scanner into the behavior you desire. 6511 6512 Sorry for the less-than-completely-satisfactory answer. 6513 6514 Vern 6515 6516 6517File: flex.info, Node: unnamed-faq-84, Next: unnamed-faq-85, Prev: unnamed-faq-83, Up: FAQ 6518 6519unnamed-faq-84 6520============== 6521 6522 To: Patrick Krusenotto <krusenot@mac-info-link.de> 6523 Subject: Re: Problems with restarting flex-2.5.2-generated scanner 6524 In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT. 6525 Date: Thu, 24 Sep 1998 23:28:43 PDT 6526 From: Vern Paxson <vern> 6527 6528 > I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately 6529 > trying to make my scanner restart with a new file after my parser stops 6530 > with a parse error. When my compiler restarts, the parser always 6531 > receives the token after the token (in the old file!) that caused the 6532 > parser error. 6533 6534 I suspect the problem is that your parser has read ahead in order 6535 to attempt to resolve an ambiguity, and when it's restarted it picks 6536 up with that token rather than reading a fresh one. If you're using 6537 yacc, then the special "error" production can sometimes be used to 6538 consume tokens in an attempt to get the parser into a consistent state. 6539 6540 Vern 6541 6542 6543File: flex.info, Node: unnamed-faq-85, Next: unnamed-faq-86, Prev: unnamed-faq-84, Up: FAQ 6544 6545unnamed-faq-85 6546============== 6547 6548 To: Henric Jungheim <junghelh@pe-nelson.com> 6549 Subject: Re: flex 2.5.4a 6550 In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST. 6551 Date: Tue, 27 Oct 1998 16:50:14 PST 6552 From: Vern Paxson <vern> 6553 6554 > This brings up a feature request: How about a command line 6555 > option to specify the filename when reading from stdin? That way one 6556 > doesn't need to create a temporary file in order to get the "#line" 6557 > directives to make sense. 6558 6559 Use -o combined with -t (per the man page description of -o). 6560 6561 > P.S., Is there any simple way to use non-blocking IO to parse multiple 6562 > streams? 6563 6564 Simple, no. 6565 6566 One approach might be to return a magic character on EWOULDBLOCK and 6567 have a rule 6568 6569 .*<magic-character> // put back .*, eat magic character 6570 6571 This is off the top of my head, not sure it'll work. 6572 6573 Vern 6574 6575 6576File: flex.info, Node: unnamed-faq-86, Next: unnamed-faq-87, Prev: unnamed-faq-85, Up: FAQ 6577 6578unnamed-faq-86 6579============== 6580 6581 To: "Repko, Billy D" <billy.d.repko@intel.com> 6582 Subject: Re: Compiling scanners 6583 In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST. 6584 Date: Thu, 14 Jan 1999 00:25:30 PST 6585 From: Vern Paxson <vern> 6586 6587 > It appears that maybe it cannot find the lfl library. 6588 6589 The Makefile in the distribution builds it, so you should have it. 6590 It's exceedingly trivial, just a main() that calls yylex() and 6591 a yyrap() that always returns 1. 6592 6593 > %% 6594 > \n ++num_lines; ++num_chars; 6595 > . ++num_chars; 6596 6597 You can't indent your rules like this - that's where the errors are coming 6598 from. Flex copies indented text to the output file, it's how you do things 6599 like 6600 6601 int num_lines_seen = 0; 6602 6603 to declare local variables. 6604 6605 Vern 6606 6607 6608File: flex.info, Node: unnamed-faq-87, Next: unnamed-faq-88, Prev: unnamed-faq-86, Up: FAQ 6609 6610unnamed-faq-87 6611============== 6612 6613 To: Erick Branderhorst <Erick.Branderhorst@asml.nl> 6614 Subject: Re: flex input buffer 6615 In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST. 6616 Date: Tue, 09 Feb 1999 21:03:37 PST 6617 From: Vern Paxson <vern> 6618 6619 > In the flex.skl file the size of the default input buffers is set. Can you 6620 > explain why this size is set and why it is such a high number. 6621 6622 It's large to optimize performance when scanning large files. You can 6623 safely make it a lot lower if needed. 6624 6625 Vern 6626 6627 6628File: flex.info, Node: unnamed-faq-88, Next: unnamed-faq-90, Prev: unnamed-faq-87, Up: FAQ 6629 6630unnamed-faq-88 6631============== 6632 6633 To: "Guido Minnen" <guidomi@cogs.susx.ac.uk> 6634 Subject: Re: Flex error message 6635 In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST. 6636 Date: Thu, 25 Feb 1999 00:11:31 PST 6637 From: Vern Paxson <vern> 6638 6639 > I'm extending a larger scanner written in Flex and I keep running into 6640 > problems. More specifically, I get the error message: 6641 > "flex: input rules are too complicated (>= 32000 NFA states)" 6642 6643 Increase the definitions in flexdef.h for: 6644 6645 #define JAMSTATE -32766 /* marks a reference to the state that always j 6646 ams */ 6647 #define MAXIMUM_MNS 31999 6648 #define BAD_SUBSCRIPT -32767 6649 6650 recompile everything, and it should all work. 6651 6652 Vern 6653 6654 6655File: flex.info, Node: unnamed-faq-90, Next: unnamed-faq-91, Prev: unnamed-faq-88, Up: FAQ 6656 6657unnamed-faq-90 6658============== 6659 6660 To: "Dmitriy Goldobin" <gold@ems.chel.su> 6661 Subject: Re: FLEX trouble 6662 In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT. 6663 Date: Tue, 01 Jun 1999 00:15:07 PDT 6664 From: Vern Paxson <vern> 6665 6666 > I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20 6667 > but rule "/*"(.|\n)*"*/" don't work ? 6668 6669 The second of these will have to scan the entire input stream (because 6670 "(.|\n)*" matches an arbitrary amount of any text) in order to see if 6671 it ends with "*/", terminating the comment. That potentially will overflow 6672 the input buffer. 6673 6674 > More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error 6675 > 'unrecognized rule'. 6676 6677 You can't use the '/' operator inside parentheses. It's not clear 6678 what "(a/b)*" actually means. 6679 6680 > I now use workaround with state <comment>, but single-rule is 6681 > better, i think. 6682 6683 Single-rule is nice but will always have the problem of either setting 6684 restrictions on comments (like not allowing multi-line comments) and/or 6685 running the risk of consuming the entire input stream, as noted above. 6686 6687 Vern 6688 6689 6690File: flex.info, Node: unnamed-faq-91, Next: unnamed-faq-92, Prev: unnamed-faq-90, Up: FAQ 6691 6692unnamed-faq-91 6693============== 6694 6695 Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18]) 6696 by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100 6697 for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT) 6698 Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999 6699 To: vern@ee.lbl.gov 6700 Date: Tue, 15 Jun 1999 08:55:43 -0700 6701 From: "Aki Niimura" <neko@my-deja.com> 6702 Message-ID: <KNONDOHDOBGAEAAA@my-deja.com> 6703 Mime-Version: 1.0 6704 Cc: 6705 X-Sent-Mail: on 6706 Reply-To: 6707 X-Mailer: MailCity Service 6708 Subject: A question on flex C++ scanner 6709 X-Sender-Ip: 12.72.207.61 6710 Organization: My Deja Email (http://www.my-deja.com:80) 6711 Content-Type: text/plain; charset=us-ascii 6712 Content-Transfer-Encoding: 7bit 6713 6714 Dear Dr. Paxon, 6715 6716 I have been using flex for years. 6717 It works very well on many projects. 6718 Most case, I used it to generate a scanner on C language. 6719 However, one project I needed to generate a scanner 6720 on C++ lanuage. Thanks to your enhancement, flex did 6721 the job. 6722 6723 Currently, I'm working on enhancing my previous project. 6724 I need to deal with multiple input streams (recursive 6725 inclusion) in this scanner (C++). 6726 I did similar thing for another scanner (C) as you 6727 explained in your documentation. 6728 6729 The generated scanner (C++) has necessary methods: 6730 - switch_to_buffer(struct yy_buffer_state *b) 6731 - yy_create_buffer(istream *is, int sz) 6732 - yy_delete_buffer(struct yy_buffer_state *b) 6733 6734 However, I couldn't figure out how to access current 6735 buffer (yy_current_buffer). 6736 6737 yy_current_buffer is a protected member of yyFlexLexer. 6738 I can't access it directly. 6739 Then, I thought yy_create_buffer() with is = 0 might 6740 return current stream buffer. But it seems not as far 6741 as I checked the source. (flex 2.5.4) 6742 6743 I went through the Web in addition to Flex documentation. 6744 However, it hasn't been successful, so far. 6745 6746 It is not my intention to bother you, but, can you 6747 comment about how to obtain the current stream buffer? 6748 6749 Your response would be highly appreciated. 6750 6751 Best regards, 6752 Aki Niimura 6753 6754 --== Sent via Deja.com http://www.deja.com/ ==-- 6755 Share what you know. Learn what you don't. 6756 6757 6758File: flex.info, Node: unnamed-faq-92, Next: unnamed-faq-93, Prev: unnamed-faq-91, Up: FAQ 6759 6760unnamed-faq-92 6761============== 6762 6763 To: neko@my-deja.com 6764 Subject: Re: A question on flex C++ scanner 6765 In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT. 6766 Date: Tue, 15 Jun 1999 09:04:24 PDT 6767 From: Vern Paxson <vern> 6768 6769 > However, I couldn't figure out how to access current 6770 > buffer (yy_current_buffer). 6771 6772 Derive your own subclass from yyFlexLexer. 6773 6774 Vern 6775 6776 6777File: flex.info, Node: unnamed-faq-93, Next: unnamed-faq-94, Prev: unnamed-faq-92, Up: FAQ 6778 6779unnamed-faq-93 6780============== 6781 6782 To: "Stones, Darren" <Darren.Stones@nectech.co.uk> 6783 Subject: Re: You're the man to see? 6784 In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT. 6785 Date: Wed, 23 Jun 1999 09:01:40 PDT 6786 From: Vern Paxson <vern> 6787 6788 > I hope you can help me. I am using Flex and Bison to produce an interpreted 6789 > language. However all goes well until I try to implement an IF statement or 6790 > a WHILE. I cannot get this to work as the parser parses all the conditions 6791 > eg. the TRUE and FALSE conditons to check for a rule match. So I cannot 6792 > make a decision!! 6793 6794 You need to use the parser to build a parse tree (= abstract syntax trwee), 6795 and when that's all done you recursively evaluate the tree, binding variables 6796 to values at that time. 6797 6798 Vern 6799 6800 6801File: flex.info, Node: unnamed-faq-94, Next: unnamed-faq-95, Prev: unnamed-faq-93, Up: FAQ 6802 6803unnamed-faq-94 6804============== 6805 6806 To: Petr Danecek <petr@ics.cas.cz> 6807 Subject: Re: flex - question 6808 In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT. 6809 Date: Fri, 02 Jul 1999 16:52:13 PDT 6810 From: Vern Paxson <vern> 6811 6812 > file, it takes an enormous amount of time. It is funny, because the 6813 > source code has only 12 rules!!! I think it looks like an exponencial 6814 > growth. 6815 6816 Right, that's the problem - some patterns (those with a lot of 6817 ambiguity, where yours has because at any given time the scanner can 6818 be in the middle of all sorts of combinations of the different 6819 rules) blow up exponentially. 6820 6821 For your rules, there is an easy fix. Change the ".*" that comes fater 6822 the directory name to "[^ ]*". With that in place, the rules are no 6823 longer nearly so ambiguous, because then once one of the directories 6824 has been matched, no other can be matched (since they all require a 6825 leading blank). 6826 6827 If that's not an acceptable solution, then you can enter a start state 6828 to pick up the .*\n after each directory is matched. 6829 6830 Also note that for speed, you'll want to add a ".*" rule at the end, 6831 otherwise rules that don't match any of the patterns will be matched 6832 very slowly, a character at a time. 6833 6834 Vern 6835 6836 6837File: flex.info, Node: unnamed-faq-95, Next: unnamed-faq-96, Prev: unnamed-faq-94, Up: FAQ 6838 6839unnamed-faq-95 6840============== 6841 6842 To: Tielman Koekemoer <tielman@spi.co.za> 6843 Subject: Re: Please help. 6844 In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT. 6845 Date: Thu, 08 Jul 1999 08:20:39 PDT 6846 From: Vern Paxson <vern> 6847 6848 > I was hoping you could help me with my problem. 6849 > 6850 > I tried compiling (gnu)flex on a Solaris 2.4 machine 6851 > but when I ran make (after configure) I got an error. 6852 > 6853 > -------------------------------------------------------------- 6854 > gcc -c -I. -I. -g -O parse.c 6855 > ./flex -t -p ./scan.l >scan.c 6856 > sh: ./flex: not found 6857 > *** Error code 1 6858 > make: Fatal error: Command failed for target `scan.c' 6859 > ------------------------------------------------------------- 6860 > 6861 > What's strange to me is that I'm only 6862 > trying to install flex now. I then edited the Makefile to 6863 > and changed where it says "FLEX = flex" to "FLEX = lex" 6864 > ( lex: the native Solaris one ) but then it complains about 6865 > the "-p" option. Is there any way I can compile flex without 6866 > using flex or lex? 6867 > 6868 > Thanks so much for your time. 6869 6870 You managed to step on the bootstrap sequence, which first copies 6871 initscan.c to scan.c in order to build flex. Try fetching a fresh 6872 distribution from ftp.ee.lbl.gov. (Or you can first try removing 6873 ".bootstrap" and doing a make again.) 6874 6875 Vern 6876 6877 6878File: flex.info, Node: unnamed-faq-96, Next: unnamed-faq-97, Prev: unnamed-faq-95, Up: FAQ 6879 6880unnamed-faq-96 6881============== 6882 6883 To: Tielman Koekemoer <tielman@spi.co.za> 6884 Subject: Re: Please help. 6885 In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT. 6886 Date: Fri, 09 Jul 1999 00:27:20 PDT 6887 From: Vern Paxson <vern> 6888 6889 > First I removed .bootstrap (and ran make) - no luck. I downloaded the 6890 > software but I still have the same problem. Is there anything else I 6891 > could try. 6892 6893 Try: 6894 6895 cp initscan.c scan.c 6896 touch scan.c 6897 make scan.o 6898 6899 If this last tries to first build scan.c from scan.l using ./flex, then 6900 your "make" is broken, in which case compile scan.c to scan.o by hand. 6901 6902 Vern 6903 6904 6905File: flex.info, Node: unnamed-faq-97, Next: unnamed-faq-98, Prev: unnamed-faq-96, Up: FAQ 6906 6907unnamed-faq-97 6908============== 6909 6910 To: Sumanth Kamenani <skamenan@crl.nmsu.edu> 6911 Subject: Re: Error 6912 In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT. 6913 Date: Tue, 20 Jul 1999 00:18:26 PDT 6914 From: Vern Paxson <vern> 6915 6916 > I am getting a compilation error. The error is given as "unknown symbol- yylex". 6917 6918 The parser relies on calling yylex(), but you're instead using the C++ scanning 6919 class, so you need to supply a yylex() "glue" function that calls an instance 6920 scanner of the scanner (e.g., "scanner->yylex()"). 6921 6922 Vern 6923 6924 6925File: flex.info, Node: unnamed-faq-98, Next: unnamed-faq-99, Prev: unnamed-faq-97, Up: FAQ 6926 6927unnamed-faq-98 6928============== 6929 6930 To: daniel@synchrods.synchrods.COM (Daniel Senderowicz) 6931 Subject: Re: lex 6932 In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST. 6933 Date: Tue, 23 Nov 1999 15:54:30 PST 6934 From: Vern Paxson <vern> 6935 6936 Well, your problem is the 6937 6938 switch (yybgin-yysvec-1) { /* witchcraft */ 6939 6940 at the beginning of lex rules. "witchcraft" == "non-portable". It's 6941 assuming knowledge of the AT&T lex's internal variables. 6942 6943 For flex, you can probably do the equivalent using a switch on YYSTATE. 6944 6945 Vern 6946 6947 6948File: flex.info, Node: unnamed-faq-99, Next: unnamed-faq-100, Prev: unnamed-faq-98, Up: FAQ 6949 6950unnamed-faq-99 6951============== 6952 6953 To: archow@hss.hns.com 6954 Subject: Re: Regarding distribution of flex and yacc based grammars 6955 In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530. 6956 Date: Wed, 22 Dec 1999 01:56:24 PST 6957 From: Vern Paxson <vern> 6958 6959 > When we provide the customer with an object code distribution, is it 6960 > necessary for us to provide source 6961 > for the generated C files from flex and bison since they are generated by 6962 > flex and bison ? 6963 6964 For flex, no. I don't know what the current state of this is for bison. 6965 6966 > Also, is there any requrirement for us to neccessarily provide source for 6967 > the grammar files which are fed into flex and bison ? 6968 6969 Again, for flex, no. 6970 6971 See the file "COPYING" in the flex distribution for the legalese. 6972 6973 Vern 6974 6975 6976File: flex.info, Node: unnamed-faq-100, Next: unnamed-faq-101, Prev: unnamed-faq-99, Up: FAQ 6977 6978unnamed-faq-100 6979=============== 6980 6981 To: Martin Gallwey <gallweym@hyperion.moe.ul.ie> 6982 Subject: Re: Flex, and self referencing rules 6983 In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST. 6984 Date: Sat, 19 Feb 2000 18:33:16 PST 6985 From: Vern Paxson <vern> 6986 6987 > However, I do not use unput anywhere. I do use self-referencing 6988 > rules like this: 6989 > 6990 > UnaryExpr ({UnionExpr})|("-"{UnaryExpr}) 6991 6992 You can't do this - flex is *not* a parser like yacc (which does indeed 6993 allow recursion), it is a scanner that's confined to regular expressions. 6994 6995 Vern 6996 6997 6998File: flex.info, Node: unnamed-faq-101, Next: What is the difference between YYLEX_PARAM and YY_DECL?, Prev: unnamed-faq-100, Up: FAQ 6999 7000unnamed-faq-101 7001=============== 7002 7003 To: slg3@lehigh.edu (SAMUEL L. GULDEN) 7004 Subject: Re: Flex problem 7005 In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST. 7006 Date: Thu, 02 Mar 2000 23:00:46 PST 7007 From: Vern Paxson <vern> 7008 7009 If this is exactly your program: 7010 7011 > digit [0-9] 7012 > digits {digit}+ 7013 > whitespace [ \t\n]+ 7014 > 7015 > %% 7016 > "[" { printf("open_brac\n");} 7017 > "]" { printf("close_brac\n");} 7018 > "+" { printf("addop\n");} 7019 > "*" { printf("multop\n");} 7020 > {digits} { printf("NUMBER = %s\n", yytext);} 7021 > whitespace ; 7022 7023 then the problem is that the last rule needs to be "{whitespace}" ! 7024 7025 Vern 7026 7027 7028File: flex.info, Node: What is the difference between YYLEX_PARAM and YY_DECL?, Next: Why do I get "conflicting types for yylex" error?, Prev: unnamed-faq-101, Up: FAQ 7029 7030What is the difference between YYLEX_PARAM and YY_DECL? 7031======================================================= 7032 7033YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to 7034pass extra params when it calls yylex() from the parser. 7035 7036 YY_DECL is the Flex declaration of yylex. The default is similar to 7037this: 7038 7039 #define int yy_lex () 7040 7041 7042File: flex.info, Node: Why do I get "conflicting types for yylex" error?, Next: How do I access the values set in a Flex action from within a Bison action?, Prev: What is the difference between YYLEX_PARAM and YY_DECL?, Up: FAQ 7043 7044Why do I get "conflicting types for yylex" error? 7045================================================= 7046 7047This is a compiler error regarding a generated Bison parser, not a Flex 7048scanner. It means you need a prototype of yylex() in the top of the 7049Bison file. Be sure the prototype matches YY_DECL. 7050 7051 7052File: flex.info, Node: How do I access the values set in a Flex action from within a Bison action?, Prev: Why do I get "conflicting types for yylex" error?, Up: FAQ 7053 7054How do I access the values set in a Flex action from within a Bison action? 7055=========================================================================== 7056 7057With $1, $2, $3, etc. These are called "Semantic Values" in the Bison 7058manual. See *note (bison)Top::. 7059 7060 7061File: flex.info, Node: Appendices, Next: Indices, Prev: FAQ, Up: Top 7062 7063Appendix A Appendices 7064********************* 7065 7066* Menu: 7067 7068* Makefiles and Flex:: 7069* Bison Bridge:: 7070* M4 Dependency:: 7071* Common Patterns:: 7072 7073 7074File: flex.info, Node: Makefiles and Flex, Next: Bison Bridge, Prev: Appendices, Up: Appendices 7075 7076A.1 Makefiles and Flex 7077====================== 7078 7079In this appendix, we provide tips for writing Makefiles to build your 7080scanners. 7081 7082 In a traditional build environment, we say that the '.c' files are 7083the sources, and the '.o' files are the intermediate files. When using 7084'flex', however, the '.l' files are the sources, and the generated '.c' 7085files (along with the '.o' files) are the intermediate files. This 7086requires you to carefully plan your Makefile. 7087 7088 Modern 'make' programs understand that 'foo.l' is intended to 7089generate 'lex.yy.c' or 'foo.c', and will behave accordingly(1)(2). The 7090following Makefile does not explicitly instruct 'make' how to build 7091'foo.c' from 'foo.l'. Instead, it relies on the implicit rules of the 7092'make' program to build the intermediate file, 'scan.c': 7093 7094 # Basic Makefile -- relies on implicit rules 7095 # Creates "myprogram" from "scan.l" and "myprogram.c" 7096 # 7097 LEX=flex 7098 myprogram: scan.o myprogram.o 7099 scan.o: scan.l 7100 7101 7102 For simple cases, the above may be sufficient. For other cases, you 7103may have to explicitly instruct 'make' how to build your scanner. The 7104following is an example of a Makefile containing explicit rules: 7105 7106 # Basic Makefile -- provides explicit rules 7107 # Creates "myprogram" from "scan.l" and "myprogram.c" 7108 # 7109 LEX=flex 7110 myprogram: scan.o myprogram.o 7111 $(CC) -o $@ $(LDFLAGS) $^ 7112 7113 myprogram.o: myprogram.c 7114 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ 7115 7116 scan.o: scan.c 7117 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ 7118 7119 scan.c: scan.l 7120 $(LEX) $(LFLAGS) -o $@ $^ 7121 7122 clean: 7123 $(RM) *.o scan.c 7124 7125 7126 Notice in the above example that 'scan.c' is in the 'clean' target. 7127This is because we consider the file 'scan.c' to be an intermediate 7128file. 7129 7130 Finally, we provide a realistic example of a 'flex' scanner used with 7131a 'bison' parser(3). There is a tricky problem we have to deal with. 7132Since a 'flex' scanner will typically include a header file (e.g., 7133'y.tab.h') generated by the parser, we need to be sure that the header 7134file is generated BEFORE the scanner is compiled. We handle this case 7135in the following example: 7136 7137 # Makefile example -- scanner and parser. 7138 # Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c" 7139 # 7140 LEX = flex 7141 YACC = bison -y 7142 YFLAGS = -d 7143 objects = scan.o parse.o myprogram.o 7144 7145 myprogram: $(objects) 7146 scan.o: scan.l parse.c 7147 parse.o: parse.y 7148 myprogram.o: myprogram.c 7149 7150 7151 In the above example, notice the line, 7152 7153 scan.o: scan.l parse.c 7154 7155 , which lists the file 'parse.c' (the generated parser) as a 7156dependency of 'scan.o'. We want to ensure that the parser is created 7157before the scanner is compiled, and the above line seems to do the 7158trick. Feel free to experiment with your specific implementation of 7159'make'. 7160 7161 For more details on writing Makefiles, see *note (make)Top::. 7162 7163 ---------- Footnotes ---------- 7164 7165 (1) GNU 'make' and GNU 'automake' are two such programs that provide 7166implicit rules for flex-generated scanners. 7167 7168 (2) GNU 'automake' may generate code to execute flex in 7169lex-compatible mode, or to stdout. If this is not what you want, then 7170you should provide an explicit rule in your Makefile.am 7171 7172 (3) This example also applies to yacc parsers. 7173 7174 7175File: flex.info, Node: Bison Bridge, Next: M4 Dependency, Prev: Makefiles and Flex, Up: Appendices 7176 7177A.2 C Scanners with Bison Parsers 7178================================= 7179 7180This section describes the 'flex' features useful when integrating 7181'flex' with 'GNU bison'(1). Skip this section if you are not using 7182'bison' with your scanner. Here we discuss only the 'flex' half of the 7183'flex' and 'bison' pair. We do not discuss 'bison' in any detail. For 7184more information about generating 'bison' parsers, see *note 7185(bison)Top::. 7186 7187 A compatible 'bison' scanner is generated by declaring '%option 7188bison-bridge' or by supplying '--bison-bridge' when invoking 'flex' from 7189the command line. This instructs 'flex' that the macro 'yylval' may be 7190used. The data type for 'yylval', 'YYSTYPE', is typically defined in a 7191header file, included in section 1 of the 'flex' input file. For a list 7192of functions and macros available, *Note bison-functions::. 7193 7194 The declaration of yylex becomes, 7195 7196 int yylex ( YYSTYPE * lvalp, yyscan_t scanner ); 7197 7198 If '%option bison-locations' is specified, then the declaration 7199becomes, 7200 7201 int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner ); 7202 7203 Note that the macros 'yylval' and 'yylloc' evaluate to pointers. 7204Support for 'yylloc' is optional in 'bison', so it is optional in 'flex' 7205as well. The following is an example of a 'flex' scanner that is 7206compatible with 'bison'. 7207 7208 /* Scanner for "C" assignment statements... sort of. */ 7209 %{ 7210 #include "y.tab.h" /* Generated by bison. */ 7211 %} 7212 7213 %option bison-bridge bison-locations 7214 % 7215 7216 [[:digit:]]+ { yylval->num = atoi(yytext); return NUMBER;} 7217 [[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;} 7218 "="|";" { return yytext[0];} 7219 . {} 7220 % 7221 7222 As you can see, there really is no magic here. We just use 'yylval' 7223as we would any other variable. The data type of 'yylval' is generated 7224by 'bison', and included in the file 'y.tab.h'. Here is the 7225corresponding 'bison' parser: 7226 7227 /* Parser to convert "C" assignments to lisp. */ 7228 %{ 7229 /* Pass the argument to yyparse through to yylex. */ 7230 #define YYPARSE_PARAM scanner 7231 #define YYLEX_PARAM scanner 7232 %} 7233 %locations 7234 %pure_parser 7235 %union { 7236 int num; 7237 char* str; 7238 } 7239 %token <str> STRING 7240 %token <num> NUMBER 7241 %% 7242 assignment: 7243 STRING '=' NUMBER ';' { 7244 printf( "(setf %s %d)", $1, $3 ); 7245 } 7246 ; 7247 7248 ---------- Footnotes ---------- 7249 7250 (1) The features described here are purely optional, and are by no 7251means the only way to use flex with bison. We merely provide some glue 7252to ease development of your parser-scanner pair. 7253 7254 7255File: flex.info, Node: M4 Dependency, Next: Common Patterns, Prev: Bison Bridge, Up: Appendices 7256 7257A.3 M4 Dependency 7258================= 7259 7260The macro processor 'm4'(1) must be installed wherever flex is 7261installed. 'flex' invokes 'm4', found by searching the directories in 7262the 'PATH' environment variable. Any code you place in section 1 or in 7263the actions will be sent through m4. Please follow these rules to 7264protect your code from unwanted 'm4' processing. 7265 7266 * Do not use symbols that begin with, 'm4_', such as, 'm4_define', or 7267 'm4_include', since those are reserved for 'm4' macro names. If 7268 for some reason you need m4_ as a prefix, use a preprocessor 7269 #define to get your symbol past m4 unmangled. 7270 7271 * Do not use the strings '[[' or ']]' anywhere in your code. The 7272 former is not valid in C, except within comments and strings, but 7273 the latter is valid in code such as 'x[y[z]]'. The solution is 7274 simple. To get the literal string '"]]"', use '"]""]"'. To get 7275 the array notation 'x[y[z]]', use 'x[y[z] ]'. Flex will attempt to 7276 detect these sequences in user code, and escape them. However, 7277 it's best to avoid this complexity where possible, by removing such 7278 sequences from your code. 7279 7280 'm4' is only required at the time you run 'flex'. The generated 7281scanner is ordinary C or C++, and does _not_ require 'm4'. 7282 7283 ---------- Footnotes ---------- 7284 7285 (1) The use of m4 is subject to change in future revisions of flex. 7286It is not part of the public API of flex. Do not depend on it. 7287 7288 7289File: flex.info, Node: Common Patterns, Prev: M4 Dependency, Up: Appendices 7290 7291A.4 Common Patterns 7292=================== 7293 7294This appendix provides examples of common regular expressions you might 7295use in your scanner. 7296 7297* Menu: 7298 7299* Numbers:: 7300* Identifiers:: 7301* Quoted Constructs:: 7302* Addresses:: 7303 7304 7305File: flex.info, Node: Numbers, Next: Identifiers, Up: Common Patterns 7306 7307A.4.1 Numbers 7308------------- 7309 7310C99 decimal constant 7311 '([[:digit:]]{-}[0])[[:digit:]]*' 7312 7313C99 hexadecimal constant 7314 '0[xX][[:xdigit:]]+' 7315 7316C99 octal constant 7317 '0[01234567]*' 7318 7319C99 floating point constant 7320 {dseq} ([[:digit:]]+) 7321 {dseq_opt} ([[:digit:]]*) 7322 {frac} (({dseq_opt}"."{dseq})|{dseq}".") 7323 {exp} ([eE][+-]?{dseq}) 7324 {exp_opt} ({exp}?) 7325 {fsuff} [flFL] 7326 {fsuff_opt} ({fsuff}?) 7327 {hpref} (0[xX]) 7328 {hdseq} ([[:xdigit:]]+) 7329 {hdseq_opt} ([[:xdigit:]]*) 7330 {hfrac} (({hdseq_opt}"."{hdseq})|({hdseq}".")) 7331 {bexp} ([pP][+-]?{dseq}) 7332 {dfc} (({frac}{exp_opt}{fsuff_opt})|({dseq}{exp}{fsuff_opt})) 7333 {hfc} (({hpref}{hfrac}{bexp}{fsuff_opt})|({hpref}{hdseq}{bexp}{fsuff_opt})) 7334 7335 {c99_floating_point_constant} ({dfc}|{hfc}) 7336 7337 See C99 section 6.4.4.2 for the gory details. 7338 7339 7340File: flex.info, Node: Identifiers, Next: Quoted Constructs, Prev: Numbers, Up: Common Patterns 7341 7342A.4.2 Identifiers 7343----------------- 7344 7345C99 Identifier 7346 ucn ((\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8}))) 7347 nondigit [_[:alpha:]] 7348 c99_id ([_[:alpha:]]|{ucn})([_[:alnum:]]|{ucn})* 7349 7350 Technically, the above pattern does not encompass all possible C99 7351 identifiers, since C99 allows for "implementation-defined" 7352 characters. In practice, C compilers follow the above pattern, 7353 with the addition of the '$' character. 7354 7355UTF-8 Encoded Unicode Code Point 7356 [\x09\x0A\x0D\x20-\x7E]|[\xC2-\xDF][\x80-\xBF]|\xE0[\xA0-\xBF][\x80-\xBF]|[\xE1-\xEC\xEE\xEF]([\x80-\xBF]{2})|\xED[\x80-\x9F][\x80-\xBF]|\xF0[\x90-\xBF]([\x80-\xBF]{2})|[\xF1-\xF3]([\x80-\xBF]{3})|\xF4[\x80-\x8F]([\x80-\xBF]{2}) 7357 7358 7359File: flex.info, Node: Quoted Constructs, Next: Addresses, Prev: Identifiers, Up: Common Patterns 7360 7361A.4.3 Quoted Constructs 7362----------------------- 7363 7364C99 String Literal 7365 'L?\"([^\"\\\n]|(\\['\"?\\abfnrtv])|(\\([0123456]{1,3}))|(\\x[[:xdigit:]]+)|(\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8})))*\"' 7366 7367C99 Comment 7368 '("/*"([^*]|"*"[^/])*"*/")|("/"(\\\n)*"/"[^\n]*)' 7369 7370 Note that in C99, a '//'-style comment may be split across lines, 7371 and, contrary to popular belief, does not include the trailing '\n' 7372 character. 7373 7374 A better way to scan '/* */' comments is by line, rather than 7375 matching possibly huge comments all at once. This will allow you 7376 to scan comments of unlimited length, as long as line breaks appear 7377 at sane intervals. This is also more efficient when used with 7378 automatic line number processing. *Note option-yylineno::. 7379 7380 <INITIAL>{ 7381 "/*" BEGIN(COMMENT); 7382 } 7383 <COMMENT>{ 7384 "*/" BEGIN(0); 7385 [^*\n]+ ; 7386 "*"[^/] ; 7387 \n ; 7388 } 7389 7390 7391File: flex.info, Node: Addresses, Prev: Quoted Constructs, Up: Common Patterns 7392 7393A.4.4 Addresses 7394--------------- 7395 7396IPv4 Address 7397 dec-octet [0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5] 7398 IPv4address {dec-octet}\.{dec-octet}\.{dec-octet}\.{dec-octet} 7399 7400IPv6 Address 7401 h16 [0-9A-Fa-f]{1,4} 7402 ls32 {h16}:{h16}|{IPv4address} 7403 IPv6address ({h16}:){6}{ls32}| 7404 ::({h16}:){5}{ls32}| 7405 ({h16})?::({h16}:){4}{ls32}| 7406 (({h16}:){0,1}{h16})?::({h16}:){3}{ls32}| 7407 (({h16}:){0,2}{h16})?::({h16}:){2}{ls32}| 7408 (({h16}:){0,3}{h16})?::{h16}:{ls32}| 7409 (({h16}:){0,4}{h16})?::{ls32}| 7410 (({h16}:){0,5}{h16})?::{h16}| 7411 (({h16}:){0,6}{h16})?:: 7412 7413 See RFC 2373 (http://www.ietf.org/rfc/rfc2373.txt) for details. 7414 Note that you have to fold the definition of 'IPv6address' into one 7415 line and that it also matches the "unspecified address" "::". 7416 7417URI 7418 '(([^:/?#]+):)?("//"([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?' 7419 7420 This pattern is nearly useless, since it allows just about any 7421 character to appear in a URI, including spaces and control 7422 characters. See RFC 2396 (http://www.ietf.org/rfc/rfc2396.txt) for 7423 details. 7424 7425 7426File: flex.info, Node: Indices, Prev: Appendices, Up: Top 7427 7428Indices 7429******* 7430 7431* Menu: 7432 7433* Concept Index:: 7434* Index of Functions and Macros:: 7435* Index of Variables:: 7436* Index of Data Types:: 7437* Index of Hooks:: 7438* Index of Scanner Options:: 7439 7440 7441File: flex.info, Node: Concept Index, Next: Index of Functions and Macros, Prev: Indices, Up: Indices 7442 7443Concept Index 7444============= 7445 7446�[index�] 7447* Menu: 7448 7449* $ as normal character in patterns: Patterns. (line 275) 7450* %array, advantages of: Matching. (line 43) 7451* %array, use of: Matching. (line 29) 7452* %array, with C++: Matching. (line 65) 7453* %option noyywrapp: Generated Scanner. (line 93) 7454* %pointer, and unput(): Actions. (line 162) 7455* %pointer, use of: Matching. (line 29) 7456* %top: Definitions Section. (line 44) 7457* %{ and %}, in Definitions Section: Definitions Section. (line 40) 7458* %{ and %}, in Rules Section: Actions. (line 26) 7459* <<EOF>>, use of: EOF. (line 33) 7460* [] in patterns: Patterns. (line 15) 7461* ^ as non-special character in patterns: Patterns. (line 275) 7462* |, in actions: Actions. (line 33) 7463* |, use of: Actions. (line 83) 7464* accessor functions, use of: Accessor Methods. (line 18) 7465* actions: Actions. (line 6) 7466* actions, embedded C strings: Actions. (line 26) 7467* actions, redefining YY_BREAK: Misc Macros. (line 49) 7468* actions, use of { and }: Actions. (line 26) 7469* aliases, how to define: Definitions Section. (line 10) 7470* arguments, command-line: Scanner Options. (line 6) 7471* array, default size for yytext: User Values. (line 13) 7472* backing up, eliminating: Performance. (line 54) 7473* backing up, eliminating by adding error rules: Performance. (line 104) 7474* backing up, eliminating with catch-all rule: Performance. (line 118) 7475* backing up, example of eliminating: Performance. (line 49) 7476* BEGIN: Actions. (line 57) 7477* BEGIN, explanation: Start Conditions. (line 84) 7478* beginning of line, in patterns: Patterns. (line 127) 7479* bison, bridging with flex: Bison Bridge. (line 6) 7480* bison, parser: Bison Bridge. (line 53) 7481* bison, scanner to be called from bison: Bison Bridge. (line 34) 7482* BOL, checking the BOL flag: Misc Macros. (line 46) 7483* BOL, in patterns: Patterns. (line 127) 7484* BOL, setting it: Misc Macros. (line 40) 7485* braces in patterns: Patterns. (line 42) 7486* bugs, reporting: Reporting Bugs. (line 6) 7487* C code in flex input: Definitions Section. (line 40) 7488* C++: Cxx. (line 9) 7489* C++ and %array: User Values. (line 23) 7490* C++ I/O, customizing: How do I use my own I/O classes in a C++ scanner?. 7491 (line 9) 7492* C++ scanners, including multiple scanners: Cxx. (line 197) 7493* C++ scanners, use of: Cxx. (line 128) 7494* c++, experimental form of scanner class: Cxx. (line 6) 7495* C++, multiple different scanners: Cxx. (line 192) 7496* C-strings, in actions: Actions. (line 26) 7497* case-insensitive, effect on character classes: Patterns. (line 216) 7498* character classes in patterns: Patterns. (line 186) 7499* character classes in patterns, syntax of: Patterns. (line 15) 7500* character classes, equivalence of: Patterns. (line 205) 7501* clearing an input buffer: Multiple Input Buffers. 7502 (line 66) 7503* command-line options: Scanner Options. (line 6) 7504* comments in flex input: Definitions Section. (line 37) 7505* comments in the input: Comments in the Input. 7506 (line 24) 7507* comments, discarding: Actions. (line 176) 7508* comments, example of scanning C comments: Start Conditions. (line 140) 7509* comments, in actions: Actions. (line 26) 7510* comments, in rules section: Comments in the Input. 7511 (line 11) 7512* comments, syntax of: Comments in the Input. 7513 (line 6) 7514* comments, valid uses of: Comments in the Input. 7515 (line 24) 7516* compressing whitespace: Actions. (line 22) 7517* concatenation, in patterns: Patterns. (line 111) 7518* copyright of flex: Copyright. (line 6) 7519* counting characters and lines: Simple Examples. (line 23) 7520* customizing I/O in C++ scanners: How do I use my own I/O classes in a C++ scanner?. 7521 (line 9) 7522* default rule: Simple Examples. (line 15) 7523* default rule <1>: Matching. (line 20) 7524* defining pattern aliases: Definitions Section. (line 21) 7525* Definitions, in flex input: Definitions Section. (line 6) 7526* deleting lines from input: Actions. (line 13) 7527* discarding C comments: Actions. (line 176) 7528* distributing flex: Copyright. (line 6) 7529* ECHO: Actions. (line 54) 7530* ECHO, and yyout: Generated Scanner. (line 101) 7531* embedding C code in flex input: Definitions Section. (line 40) 7532* end of file, in patterns: Patterns. (line 150) 7533* end of line, in negated character classes: Patterns. (line 237) 7534* end of line, in patterns: Patterns. (line 131) 7535* end-of-file, and yyrestart(): Generated Scanner. (line 42) 7536* EOF and yyrestart(): Generated Scanner. (line 42) 7537* EOF in patterns, syntax of: Patterns. (line 150) 7538* EOF, example using multiple input buffers: Multiple Input Buffers. 7539 (line 81) 7540* EOF, explanation: EOF. (line 6) 7541* EOF, pushing back: Actions. (line 170) 7542* EOL, in negated character classes: Patterns. (line 237) 7543* EOL, in patterns: Patterns. (line 131) 7544* error messages, end of buffer missed: Lex and Posix. (line 50) 7545* error reporting, diagnostic messages: Diagnostics. (line 6) 7546* error reporting, in C++: Cxx. (line 112) 7547* error rules, to eliminate backing up: Performance. (line 102) 7548* escape sequences in patterns, syntax of: Patterns. (line 57) 7549* exiting with yyterminate(): Actions. (line 212) 7550* experimental form of c++ scanner class: Cxx. (line 6) 7551* extended scope of start conditions: Start Conditions. (line 270) 7552* file format: Format. (line 6) 7553* file format, serialized tables: Tables File Format. (line 6) 7554* flushing an input buffer: Multiple Input Buffers. 7555 (line 66) 7556* flushing the internal buffer: Actions. (line 206) 7557* format of flex input: Format. (line 6) 7558* format of input file: Format. (line 9) 7559* freeing tables: Loading and Unloading Serialized Tables. 7560 (line 6) 7561* getting current start state with YY_START: Start Conditions. 7562 (line 189) 7563* halting with yyterminate(): Actions. (line 212) 7564* handling include files with multiple input buffers: Multiple Input Buffers. 7565 (line 87) 7566* handling include files with multiple input buffers <1>: Multiple Input Buffers. 7567 (line 122) 7568* header files, with C++: Cxx. (line 197) 7569* include files, with C++: Cxx. (line 197) 7570* input file, Definitions section: Definitions Section. (line 6) 7571* input file, Rules Section: Rules Section. (line 6) 7572* input file, user code Section: User Code Section. (line 6) 7573* input(): Actions. (line 173) 7574* input(), and C++: Actions. (line 202) 7575* input, format of: Format. (line 6) 7576* input, matching: Matching. (line 6) 7577* keywords, for performance: Performance. (line 200) 7578* lex (traditional) and POSIX: Lex and Posix. (line 6) 7579* LexerInput, overriding: How do I use my own I/O classes in a C++ scanner?. 7580 (line 9) 7581* LexerOutput, overriding: How do I use my own I/O classes in a C++ scanner?. 7582 (line 9) 7583* limitations of flex: Limitations. (line 6) 7584* literal text in patterns, syntax of: Patterns. (line 54) 7585* loading tables at runtime: Loading and Unloading Serialized Tables. 7586 (line 6) 7587* m4: M4 Dependency. (line 6) 7588* Makefile, example of implicit rules: Makefiles and Flex. (line 21) 7589* Makefile, explicit example: Makefiles and Flex. (line 33) 7590* Makefile, syntax: Makefiles and Flex. (line 6) 7591* matching C-style double-quoted strings: Start Conditions. (line 203) 7592* matching, and trailing context: Matching. (line 6) 7593* matching, length of: Matching. (line 6) 7594* matching, multiple matches: Matching. (line 6) 7595* member functions, C++: Cxx. (line 9) 7596* memory management: Memory Management. (line 6) 7597* memory, allocating input buffers: Multiple Input Buffers. 7598 (line 19) 7599* memory, considerations for reentrant scanners: Init and Destroy Functions. 7600 (line 6) 7601* memory, deleting input buffers: Multiple Input Buffers. 7602 (line 46) 7603* memory, for start condition stacks: Start Conditions. (line 301) 7604* memory, serialized tables: Serialized Tables. (line 6) 7605* memory, serialized tables <1>: Loading and Unloading Serialized Tables. 7606 (line 6) 7607* methods, c++: Cxx. (line 9) 7608* minimal scanner: Matching. (line 24) 7609* multiple input streams: Multiple Input Buffers. 7610 (line 6) 7611* name definitions, not POSIX: Lex and Posix. (line 75) 7612* negating ranges in patterns: Patterns. (line 23) 7613* newline, matching in patterns: Patterns. (line 135) 7614* non-POSIX features of flex: Lex and Posix. (line 142) 7615* noyywrap, %option: Generated Scanner. (line 93) 7616* NULL character in patterns, syntax of: Patterns. (line 62) 7617* octal characters in patterns: Patterns. (line 65) 7618* options, command-line: Scanner Options. (line 6) 7619* overriding LexerInput: How do I use my own I/O classes in a C++ scanner?. 7620 (line 9) 7621* overriding LexerOutput: How do I use my own I/O classes in a C++ scanner?. 7622 (line 9) 7623* overriding the memory routines: Overriding The Default Memory Management. 7624 (line 38) 7625* Pascal-like language: Simple Examples. (line 49) 7626* pattern aliases, defining: Definitions Section. (line 21) 7627* pattern aliases, expansion of: Patterns. (line 51) 7628* pattern aliases, how to define: Definitions Section. (line 10) 7629* pattern aliases, use of: Definitions Section. (line 28) 7630* patterns and actions on different lines: Lex and Posix. (line 101) 7631* patterns, character class equivalence: Patterns. (line 205) 7632* patterns, common: Common Patterns. (line 6) 7633* patterns, end of line: Patterns. (line 300) 7634* patterns, grouping and precedence: Patterns. (line 167) 7635* patterns, in rules section: Patterns. (line 6) 7636* patterns, invalid trailing context: Patterns. (line 285) 7637* patterns, matching: Matching. (line 6) 7638* patterns, precedence of operators: Patterns. (line 161) 7639* patterns, repetitions with grouping: Patterns. (line 184) 7640* patterns, special characters treated as non-special: Patterns. 7641 (line 293) 7642* patterns, syntax: Patterns. (line 9) 7643* patterns, syntax <1>: Patterns. (line 9) 7644* patterns, tuning for performance: Performance. (line 49) 7645* patterns, valid character classes: Patterns. (line 192) 7646* performance optimization, matching longer tokens: Performance. 7647 (line 167) 7648* performance optimization, recognizing keywords: Performance. 7649 (line 205) 7650* performance, backing up: Performance. (line 49) 7651* performance, considerations: Performance. (line 6) 7652* performance, using keywords: Performance. (line 200) 7653* popping an input buffer: Multiple Input Buffers. 7654 (line 60) 7655* POSIX and lex: Lex and Posix. (line 6) 7656* POSIX comp;compliance: Lex and Posix. (line 142) 7657* POSIX, character classes in patterns, syntax of: Patterns. (line 15) 7658* preprocessor macros, for use in actions: Actions. (line 50) 7659* pushing an input buffer: Multiple Input Buffers. 7660 (line 52) 7661* pushing back characters with unput: Actions. (line 143) 7662* pushing back characters with unput(): Actions. (line 147) 7663* pushing back characters with yyless: Actions. (line 131) 7664* pushing back EOF: Actions. (line 170) 7665* ranges in patterns: Patterns. (line 19) 7666* ranges in patterns, negating: Patterns. (line 23) 7667* recognizing C comments: Start Conditions. (line 143) 7668* reentrant scanners, multiple interleaved scanners: Reentrant Uses. 7669 (line 10) 7670* reentrant scanners, recursive invocation: Reentrant Uses. (line 30) 7671* reentrant, accessing flex variables: Global Replacement. (line 6) 7672* reentrant, accessor functions: Accessor Methods. (line 6) 7673* reentrant, API explanation: Reentrant Overview. (line 6) 7674* reentrant, calling functions: Extra Reentrant Argument. 7675 (line 6) 7676* reentrant, example of: Reentrant Example. (line 6) 7677* reentrant, explanation: Reentrant. (line 6) 7678* reentrant, extra data: Extra Data. (line 6) 7679* reentrant, initialization: Init and Destroy Functions. 7680 (line 6) 7681* regular expressions, in patterns: Patterns. (line 6) 7682* REJECT: Actions. (line 61) 7683* REJECT, calling multiple times: Actions. (line 83) 7684* REJECT, performance costs: Performance. (line 12) 7685* reporting bugs: Reporting Bugs. (line 6) 7686* restarting the scanner: Lex and Posix. (line 54) 7687* RETURN, within actions: Generated Scanner. (line 57) 7688* rules, default: Simple Examples. (line 15) 7689* rules, in flex input: Rules Section. (line 6) 7690* scanner, definition of: Introduction. (line 6) 7691* sections of flex input: Format. (line 6) 7692* serialization: Serialized Tables. (line 6) 7693* serialization of tables: Creating Serialized Tables. 7694 (line 6) 7695* serialized tables, multiple scanners: Creating Serialized Tables. 7696 (line 26) 7697* stack, input buffer pop: Multiple Input Buffers. 7698 (line 60) 7699* stack, input buffer push: Multiple Input Buffers. 7700 (line 52) 7701* stacks, routines for manipulating: Start Conditions. (line 286) 7702* start condition, applying to multiple patterns: Start Conditions. 7703 (line 258) 7704* start conditions: Start Conditions. (line 6) 7705* start conditions, behavior of default rule: Start Conditions. 7706 (line 82) 7707* start conditions, exclusive: Start Conditions. (line 53) 7708* start conditions, for different interpretations of same input: Start Conditions. 7709 (line 112) 7710* start conditions, in patterns: Patterns. (line 140) 7711* start conditions, inclusive: Start Conditions. (line 44) 7712* start conditions, inclusive v.s. exclusive: Start Conditions. 7713 (line 24) 7714* start conditions, integer values: Start Conditions. (line 163) 7715* start conditions, multiple: Start Conditions. (line 17) 7716* start conditions, special wildcard condition: Start Conditions. 7717 (line 68) 7718* start conditions, use of a stack: Start Conditions. (line 286) 7719* start conditions, use of wildcard condition (<*>): Start Conditions. 7720 (line 72) 7721* start conditions, using BEGIN: Start Conditions. (line 95) 7722* stdin, default for yyin: Generated Scanner. (line 37) 7723* stdout, as default for yyout: Generated Scanner. (line 101) 7724* strings, scanning strings instead of files: Multiple Input Buffers. 7725 (line 175) 7726* tables, creating serialized: Creating Serialized Tables. 7727 (line 6) 7728* tables, file format: Tables File Format. (line 6) 7729* tables, freeing: Loading and Unloading Serialized Tables. 7730 (line 6) 7731* tables, loading and unloading: Loading and Unloading Serialized Tables. 7732 (line 6) 7733* terminating with yyterminate(): Actions. (line 212) 7734* token: Matching. (line 14) 7735* trailing context, in patterns: Patterns. (line 118) 7736* trailing context, limits of: Patterns. (line 275) 7737* trailing context, matching: Matching. (line 6) 7738* trailing context, performance costs: Performance. (line 12) 7739* trailing context, variable length: Performance. (line 141) 7740* unput(): Actions. (line 143) 7741* unput(), and %pointer: Actions. (line 162) 7742* unput(), pushing back characters: Actions. (line 147) 7743* user code, in flex input: User Code Section. (line 6) 7744* username expansion: Simple Examples. (line 8) 7745* using integer values of start condition names: Start Conditions. 7746 (line 163) 7747* verbatim text in patterns, syntax of: Patterns. (line 54) 7748* warning, dangerous trailing context: Limitations. (line 20) 7749* warning, rule cannot be matched: Diagnostics. (line 14) 7750* warnings, diagnostic messages: Diagnostics. (line 6) 7751* whitespace, compressing: Actions. (line 22) 7752* yacc interface: Yacc. (line 17) 7753* yacc, interface: Yacc. (line 6) 7754* yyalloc, overriding: Overriding The Default Memory Management. 7755 (line 6) 7756* yyfree, overriding: Overriding The Default Memory Management. 7757 (line 6) 7758* yyin: Generated Scanner. (line 37) 7759* yyinput(): Actions. (line 202) 7760* yyleng: Matching. (line 14) 7761* yyleng, modification of: Actions. (line 47) 7762* yyless(): Actions. (line 125) 7763* yyless(), pushing back characters: Actions. (line 131) 7764* yylex(), in generated scanner: Generated Scanner. (line 6) 7765* yylex(), overriding: Generated Scanner. (line 16) 7766* yylex, overriding the prototype of: Generated Scanner. (line 20) 7767* yylineno, in a reentrant scanner: Reentrant Functions. (line 36) 7768* yylineno, performance costs: Performance. (line 12) 7769* yymore(): Actions. (line 104) 7770* yymore() to append token to previous token: Actions. (line 110) 7771* yymore(), mega-kludge: Actions. (line 110) 7772* yymore, and yyleng: Actions. (line 47) 7773* yymore, performance penalty of: Actions. (line 119) 7774* yyout: Generated Scanner. (line 101) 7775* yyrealloc, overriding: Overriding The Default Memory Management. 7776 (line 6) 7777* yyrestart(): Generated Scanner. (line 42) 7778* yyterminate(): Actions. (line 212) 7779* yytext: Matching. (line 14) 7780* yytext, default array size: User Values. (line 13) 7781* yytext, memory considerations: A Note About yytext And Memory. 7782 (line 6) 7783* yytext, modification of: Actions. (line 42) 7784* yytext, two types of: Matching. (line 29) 7785* yywrap(): Generated Scanner. (line 85) 7786* yywrap, default for: Generated Scanner. (line 93) 7787* YY_CURRENT_BUFFER, and multiple buffers Finally, the macro: Multiple Input Buffers. 7788 (line 78) 7789* YY_EXTRA_TYPE, defining your own type: Extra Data. (line 33) 7790* YY_FLUSH_BUFFER: Actions. (line 206) 7791* YY_INPUT: Generated Scanner. (line 61) 7792* YY_INPUT, overriding: Generated Scanner. (line 71) 7793* YY_START, example: Start Conditions. (line 185) 7794* YY_USER_ACTION to track each time a rule is matched: Misc Macros. 7795 (line 14) 7796 7797