1# Stream 2 3In RapidJSON, `rapidjson::Stream` is a concept for reading/writing JSON. Here we first show how to use streams provided. And then see how to create a custom stream. 4 5[TOC] 6 7# Memory Streams {#MemoryStreams} 8 9Memory streams store JSON in memory. 10 11## StringStream (Input) {#StringStream} 12 13`StringStream` is the most basic input stream. It represents a complete, read-only JSON stored in memory. It is defined in `rapidjson/rapidjson.h`. 14 15~~~~~~~~~~cpp 16#include "rapidjson/document.h" // will include "rapidjson/rapidjson.h" 17 18using namespace rapidjson; 19 20// ... 21const char json[] = "[1, 2, 3, 4]"; 22StringStream s(json); 23 24Document d; 25d.ParseStream(s); 26~~~~~~~~~~ 27 28Since this is very common usage, `Document::Parse(const char*)` is provided to do exactly the same as above: 29 30~~~~~~~~~~cpp 31// ... 32const char json[] = "[1, 2, 3, 4]"; 33Document d; 34d.Parse(json); 35~~~~~~~~~~ 36 37Note that, `StringStream` is a typedef of `GenericStringStream<UTF8<> >`, user may use another encodings to represent the character set of the stream. 38 39## StringBuffer (Output) {#StringBuffer} 40 41`StringBuffer` is a simple output stream. It allocates a memory buffer for writing the whole JSON. Use `GetString()` to obtain the buffer. 42 43~~~~~~~~~~cpp 44#include "rapidjson/stringbuffer.h" 45 46StringBuffer buffer; 47Writer<StringBuffer> writer(buffer); 48d.Accept(writer); 49 50const char* output = buffer.GetString(); 51~~~~~~~~~~ 52 53When the buffer is full, it will increases the capacity automatically. The default capacity is 256 characters (256 bytes for UTF8, 512 bytes for UTF16, etc.). User can provide an allocator and a initial capacity. 54 55~~~~~~~~~~cpp 56StringBuffer buffer1(0, 1024); // Use its allocator, initial size = 1024 57StringBuffer buffer2(allocator, 1024); 58~~~~~~~~~~ 59 60By default, `StringBuffer` will instantiate an internal allocator. 61 62Similarly, `StringBuffer` is a typedef of `GenericStringBuffer<UTF8<> >`. 63 64# File Streams {#FileStreams} 65 66When parsing a JSON from file, you may read the whole JSON into memory and use ``StringStream`` above. 67 68However, if the JSON is big, or memory is limited, you can use `FileReadStream`. It only read a part of JSON from file into buffer, and then let the part be parsed. If it runs out of characters in the buffer, it will read the next part from file. 69 70## FileReadStream (Input) {#FileReadStream} 71 72`FileReadStream` reads the file via a `FILE` pointer. And user need to provide a buffer. 73 74~~~~~~~~~~cpp 75#include "rapidjson/filereadstream.h" 76#include <cstdio> 77 78using namespace rapidjson; 79 80FILE* fp = fopen("big.json", "rb"); // non-Windows use "r" 81 82char readBuffer[65536]; 83FileReadStream is(fp, readBuffer, sizeof(readBuffer)); 84 85Document d; 86d.ParseStream(is); 87 88fclose(fp); 89~~~~~~~~~~ 90 91Different from string streams, `FileReadStream` is byte stream. It does not handle encodings. If the file is not UTF-8, the byte stream can be wrapped in a `EncodedInputStream`. It will be discussed very soon. 92 93Apart from reading file, user can also use `FileReadStream` to read `stdin`. 94 95## FileWriteStream (Output) {#FileWriteStream} 96 97`FileWriteStream` is buffered output stream. Its usage is very similar to `FileReadStream`. 98 99~~~~~~~~~~cpp 100#include "rapidjson/filewritestream.h" 101#include <cstdio> 102 103using namespace rapidjson; 104 105Document d; 106d.Parse(json); 107// ... 108 109FILE* fp = fopen("output.json", "wb"); // non-Windows use "w" 110 111char writeBuffer[65536]; 112FileWriteStream os(fp, writeBuffer, sizeof(writeBuffer)); 113 114Writer<FileWriteStream> writer(os); 115d.Accept(writer); 116 117fclose(fp); 118~~~~~~~~~~ 119 120It can also directs the output to `stdout`. 121 122# iostream Wrapper {#iostreamWrapper} 123 124Due to users' requests, RapidJSON provided official wrappers for `std::basic_istream` and `std::basic_ostream`. However, please note that the performance will be much lower than the other streams above. 125 126## IStreamWrapper {#IStreamWrapper} 127 128`IStreamWrapper` wraps any class drived from `std::istream`, such as `std::istringstream`, `std::stringstream`, `std::ifstream`, `std::fstream`, into RapidJSON's input stream. 129 130~~~cpp 131#include <rapidjson/document.h> 132#include <rapidjson/istreamwrapper.h> 133#include <fstream> 134 135using namespace rapidjson; 136using namespace std; 137 138ifstream ifs("test.json"); 139IStreamWrapper isw(ifs); 140 141Document d; 142d.ParseStream(isw); 143~~~ 144 145For classes derived from `std::wistream`, use `WIStreamWrapper`. 146 147## OStreamWrapper {#OStreamWrapper} 148 149Similarly, `OStreamWrapper` wraps any class derived from `std::ostream`, such as `std::ostringstream`, `std::stringstream`, `std::ofstream`, `std::fstream`, into RapidJSON's input stream. 150 151~~~cpp 152#include <rapidjson/document.h> 153#include <rapidjson/ostreamwrapper.h> 154#include <rapidjson/writer.h> 155#include <fstream> 156 157using namespace rapidjson; 158using namespace std; 159 160Document d; 161d.Parse(json); 162 163// ... 164 165ofstream ofs("output.json"); 166OStreamWrapper osw(ofs); 167 168Writer<OStreamWrapper> writer(osw); 169d.Accept(writer); 170~~~ 171 172For classes derived from `std::wostream`, use `WOStreamWrapper`. 173 174# Encoded Streams {#EncodedStreams} 175 176Encoded streams do not contain JSON itself, but they wrap byte streams to provide basic encoding/decoding function. 177 178As mentioned above, UTF-8 byte streams can be read directly. However, UTF-16 and UTF-32 have endian issue. To handle endian correctly, it needs to convert bytes into characters (e.g. `wchar_t` for UTF-16) while reading, and characters into bytes while writing. 179 180Besides, it also need to handle [byte order mark (BOM)](http://en.wikipedia.org/wiki/Byte_order_mark). When reading from a byte stream, it is needed to detect or just consume the BOM if exists. When writing to a byte stream, it can optionally write BOM. 181 182If the encoding of stream is known in compile-time, you may use `EncodedInputStream` and `EncodedOutputStream`. If the stream can be UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE JSON, and it is only known in runtime, you may use `AutoUTFInputStream` and `AutoUTFOutputStream`. These streams are defined in `rapidjson/encodedstream.h`. 183 184Note that, these encoded streams can be applied to streams other than file. For example, you may have a file in memory, or a custom byte stream, be wrapped in encoded streams. 185 186## EncodedInputStream {#EncodedInputStream} 187 188`EncodedInputStream` has two template parameters. The first one is a `Encoding` class, such as `UTF8`, `UTF16LE`, defined in `rapidjson/encodings.h`. The second one is the class of stream to be wrapped. 189 190~~~~~~~~~~cpp 191#include "rapidjson/document.h" 192#include "rapidjson/filereadstream.h" // FileReadStream 193#include "rapidjson/encodedstream.h" // EncodedInputStream 194#include <cstdio> 195 196using namespace rapidjson; 197 198FILE* fp = fopen("utf16le.json", "rb"); // non-Windows use "r" 199 200char readBuffer[256]; 201FileReadStream bis(fp, readBuffer, sizeof(readBuffer)); 202 203EncodedInputStream<UTF16LE<>, FileReadStream> eis(bis); // wraps bis into eis 204 205Document d; // Document is GenericDocument<UTF8<> > 206d.ParseStream<0, UTF16LE<> >(eis); // Parses UTF-16LE file into UTF-8 in memory 207 208fclose(fp); 209~~~~~~~~~~ 210 211## EncodedOutputStream {#EncodedOutputStream} 212 213`EncodedOutputStream` is similar but it has a `bool putBOM` parameter in the constructor, controlling whether to write BOM into output byte stream. 214 215~~~~~~~~~~cpp 216#include "rapidjson/filewritestream.h" // FileWriteStream 217#include "rapidjson/encodedstream.h" // EncodedOutputStream 218#include <cstdio> 219 220Document d; // Document is GenericDocument<UTF8<> > 221// ... 222 223FILE* fp = fopen("output_utf32le.json", "wb"); // non-Windows use "w" 224 225char writeBuffer[256]; 226FileWriteStream bos(fp, writeBuffer, sizeof(writeBuffer)); 227 228typedef EncodedOutputStream<UTF32LE<>, FileWriteStream> OutputStream; 229OutputStream eos(bos, true); // Write BOM 230 231Writer<OutputStream, UTF32LE<>, UTF8<>> writer(eos); 232d.Accept(writer); // This generates UTF32-LE file from UTF-8 in memory 233 234fclose(fp); 235~~~~~~~~~~ 236 237## AutoUTFInputStream {#AutoUTFInputStream} 238 239Sometimes an application may want to handle all supported JSON encoding. `AutoUTFInputStream` will detection encoding by BOM first. If BOM is unavailable, it will use characteristics of valid JSON to make detection. If neither method success, it falls back to the UTF type provided in constructor. 240 241Since the characters (code units) may be 8-bit, 16-bit or 32-bit. `AutoUTFInputStream` requires a character type which can hold at least 32-bit. We may use `unsigned`, as in the template parameter: 242 243~~~~~~~~~~cpp 244#include "rapidjson/document.h" 245#include "rapidjson/filereadstream.h" // FileReadStream 246#include "rapidjson/encodedstream.h" // AutoUTFInputStream 247#include <cstdio> 248 249using namespace rapidjson; 250 251FILE* fp = fopen("any.json", "rb"); // non-Windows use "r" 252 253char readBuffer[256]; 254FileReadStream bis(fp, readBuffer, sizeof(readBuffer)); 255 256AutoUTFInputStream<unsigned, FileReadStream> eis(bis); // wraps bis into eis 257 258Document d; // Document is GenericDocument<UTF8<> > 259d.ParseStream<0, AutoUTF<unsigned> >(eis); // This parses any UTF file into UTF-8 in memory 260 261fclose(fp); 262~~~~~~~~~~ 263 264When specifying the encoding of stream, uses `AutoUTF<CharType>` as in `ParseStream()` above. 265 266You can obtain the type of UTF via `UTFType GetType()`. And check whether a BOM is found by `HasBOM()` 267 268## AutoUTFOutputStream {#AutoUTFOutputStream} 269 270Similarly, to choose encoding for output during runtime, we can use `AutoUTFOutputStream`. This class is not automatic *per se*. You need to specify the UTF type and whether to write BOM in runtime. 271 272~~~~~~~~~~cpp 273using namespace rapidjson; 274 275void WriteJSONFile(FILE* fp, UTFType type, bool putBOM, const Document& d) { 276 char writeBuffer[256]; 277 FileWriteStream bos(fp, writeBuffer, sizeof(writeBuffer)); 278 279 typedef AutoUTFOutputStream<unsigned, FileWriteStream> OutputStream; 280 OutputStream eos(bos, type, putBOM); 281 282 Writer<OutputStream, UTF8<>, AutoUTF<> > writer; 283 d.Accept(writer); 284} 285~~~~~~~~~~ 286 287`AutoUTFInputStream` and `AutoUTFOutputStream` is more convenient than `EncodedInputStream` and `EncodedOutputStream`. They just incur a little bit runtime overheads. 288 289# Custom Stream {#CustomStream} 290 291In addition to memory/file streams, user can create their own stream classes which fits RapidJSON's API. For example, you may create network stream, stream from compressed file, etc. 292 293RapidJSON combines different types using templates. A class containing all required interface can be a stream. The Stream interface is defined in comments of `rapidjson/rapidjson.h`: 294 295~~~~~~~~~~cpp 296concept Stream { 297 typename Ch; //!< Character type of the stream. 298 299 //! Read the current character from stream without moving the read cursor. 300 Ch Peek() const; 301 302 //! Read the current character from stream and moving the read cursor to next character. 303 Ch Take(); 304 305 //! Get the current read cursor. 306 //! \return Number of characters read from start. 307 size_t Tell(); 308 309 //! Begin writing operation at the current read pointer. 310 //! \return The begin writer pointer. 311 Ch* PutBegin(); 312 313 //! Write a character. 314 void Put(Ch c); 315 316 //! Flush the buffer. 317 void Flush(); 318 319 //! End the writing operation. 320 //! \param begin The begin write pointer returned by PutBegin(). 321 //! \return Number of characters written. 322 size_t PutEnd(Ch* begin); 323} 324~~~~~~~~~~ 325 326For input stream, they must implement `Peek()`, `Take()` and `Tell()`. 327For output stream, they must implement `Put()` and `Flush()`. 328There are two special interface, `PutBegin()` and `PutEnd()`, which are only for *in situ* parsing. Normal streams do not implement them. However, if the interface is not needed for a particular stream, it is still need to a dummy implementation, otherwise will generate compilation error. 329 330## Example: istream wrapper {#ExampleIStreamWrapper} 331 332The following example is a simple wrapper of `std::istream`, which only implements 3 functions. 333 334~~~~~~~~~~cpp 335class MyIStreamWrapper { 336public: 337 typedef char Ch; 338 339 MyIStreamWrapper(std::istream& is) : is_(is) { 340 } 341 342 Ch Peek() const { // 1 343 int c = is_.peek(); 344 return c == std::char_traits<char>::eof() ? '\0' : (Ch)c; 345 } 346 347 Ch Take() { // 2 348 int c = is_.get(); 349 return c == std::char_traits<char>::eof() ? '\0' : (Ch)c; 350 } 351 352 size_t Tell() const { return (size_t)is_.tellg(); } // 3 353 354 Ch* PutBegin() { assert(false); return 0; } 355 void Put(Ch) { assert(false); } 356 void Flush() { assert(false); } 357 size_t PutEnd(Ch*) { assert(false); return 0; } 358 359private: 360 MyIStreamWrapper(const MyIStreamWrapper&); 361 MyIStreamWrapper& operator=(const MyIStreamWrapper&); 362 363 std::istream& is_; 364}; 365~~~~~~~~~~ 366 367User can use it to wrap instances of `std::stringstream`, `std::ifstream`. 368 369~~~~~~~~~~cpp 370const char* json = "[1,2,3,4]"; 371std::stringstream ss(json); 372MyIStreamWrapper is(ss); 373 374Document d; 375d.ParseStream(is); 376~~~~~~~~~~ 377 378Note that, this implementation may not be as efficient as RapidJSON's memory or file streams, due to internal overheads of the standard library. 379 380## Example: ostream wrapper {#ExampleOStreamWrapper} 381 382The following example is a simple wrapper of `std::istream`, which only implements 2 functions. 383 384~~~~~~~~~~cpp 385class MyOStreamWrapper { 386public: 387 typedef char Ch; 388 389 MyOStreamWrapper(std::ostream& os) : os_(os) { 390 } 391 392 Ch Peek() const { assert(false); return '\0'; } 393 Ch Take() { assert(false); return '\0'; } 394 size_t Tell() const { } 395 396 Ch* PutBegin() { assert(false); return 0; } 397 void Put(Ch c) { os_.put(c); } // 1 398 void Flush() { os_.flush(); } // 2 399 size_t PutEnd(Ch*) { assert(false); return 0; } 400 401private: 402 MyOStreamWrapper(const MyOStreamWrapper&); 403 MyOStreamWrapper& operator=(const MyOStreamWrapper&); 404 405 std::ostream& os_; 406}; 407~~~~~~~~~~ 408 409User can use it to wrap instances of `std::stringstream`, `std::ofstream`. 410 411~~~~~~~~~~cpp 412Document d; 413// ... 414 415std::stringstream ss; 416MyOStreamWrapper os(ss); 417 418Writer<MyOStreamWrapper> writer(os); 419d.Accept(writer); 420~~~~~~~~~~ 421 422Note that, this implementation may not be as efficient as RapidJSON's memory or file streams, due to internal overheads of the standard library. 423 424# Summary {#Summary} 425 426This section describes stream classes available in RapidJSON. Memory streams are simple. File stream can reduce the memory required during JSON parsing and generation, if the JSON is stored in file system. Encoded streams converts between byte streams and character streams. Finally, user may create custom streams using a simple interface. 427