1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> 2 3<html> 4<head> 5 <meta http-equiv="Content-Language" content="en-us"> 6 <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 7 <meta name="GENERATOR" content="Microsoft FrontPage 6.0"> 8 <meta name="ProgId" content="FrontPage.Editor.Document"> 9 10 <title>Introduction</title> 11</head> 12 13<body bgcolor="#FFFFFF"> 14 <p><img src="../../boost.png" alt="C++ Boost" width="277" height= 15 "86"/><br></p> 16 17 <h1 align="center">Introduction</h1> 18 19 <p align="left">The Boost Tokenizer package provides a flexible and 20 easy-to-use way to break a string or other character sequence into a series 21 of tokens. Below is a simple example that will break up a phrase into 22 words.</p> 23 24 <div align="left"> 25 <pre> 26// simple_example_1.cpp 27#include<iostream> 28#include<boost/tokenizer.hpp> 29#include<string> 30 31int main(){ 32 using namespace std; 33 using namespace boost; 34 string s = "This is, a test"; 35 tokenizer<> tok(s); 36 for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){ 37 cout << *beg << "\n"; 38 } 39} 40</pre> 41 </div> 42 43 <p align="left">You can choose how the string gets parsed by using the 44 TokenizerFunction. If you do not specify anything, the default 45 TokenizerFunction is <em>char_delimiters_separator<char></em> which 46 defaults to breaking up a string based on space and punctuation. Here is an 47 example using another TokenizerFunction called 48 <em>escaped_list_separator</em>. This TokenizerFunction parses a superset 49 of comma-separated value (CSV) lines. The format looks like this:</p> 50 51 <p align="left">Field 1,"putting quotes around fields, allows commas",Field 52 3</p> 53 54 <p align="left">Below is an example that will break the previous line into 55 its three fields.</p> 56 57 <div align="left"> 58 <pre> 59// simple_example_2.cpp 60#include<iostream> 61#include<boost/tokenizer.hpp> 62#include<string> 63 64int main(){ 65 using namespace std; 66 using namespace boost; 67 string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3"; 68 tokenizer<escaped_list_separator<char> > tok(s); 69 for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){ 70 cout << *beg << "\n"; 71 } 72} 73</pre> 74 </div> 75 76 <p align="left">Finally, for some TokenizerFunctions you have to pass 77 something into the constructor in order to do anything interesting. An 78 example is the offset_separator. This class breaks a string into tokens based 79 on offsets. For example, when <em>12252001</em> is parsed using offsets of 80 2,2,4 it becomes <em>12 25 2001</em>. Below is the code used.</p> 81 82 <div align="left"> 83 <pre> 84// simple_example_3.cpp 85#include<iostream> 86#include<boost/tokenizer.hpp> 87#include<string> 88 89int main(){ 90 using namespace std; 91 using namespace boost; 92 string s = "12252001"; 93 int offsets[] = {2,2,4}; 94 offset_separator f(offsets, offsets+3); 95 tokenizer<offset_separator> tok(s,f); 96 for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){ 97 cout << *beg << "\n"; 98 } 99} 100</pre> 101 </div> 102 103 <p align="left"> </p> 104 <hr> 105 106 <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src= 107 "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional" 108 height="31" width="88"></a></p> 109 110 <p>Revised 111 <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B %Y" startspan -->9 June 2010<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p> 112 113 <p><i>Copyright © 2001 John R. Bandela</i></p> 114 115 <p><i>Distributed under the Boost Software License, Version 1.0. (See 116 accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or 117 copy at <a href= 118 "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p> 119</body> 120</html> 121
