1## Beginners Guide to Contributing to Fluent Bit 2 3Assuming you have some basic knowledge of C, this guide should help you understand how to make code 4changes to Fluent Bit. 5 6### Table of Contents 7- [Libraries](#libraries) 8 - [Memory Management](#memory-management) 9 - [Strings](#strings) 10 - [HTTP Client](#http-client) 11 - [Linked Lists](#linked-lists) 12 - [Message Pack](#message-pack) 13- [Concurrency](#concurrency) 14- [Plugin API](#plugin-api) 15 - [Input](#input) 16 - [Filter](#filter) 17 - [Output](#output) 18 - [Config Maps](#config-maps) 19- [Testing](#testing) 20 - [Valgrind](#valgrind) 21- [Need more help?](#need-more-help) 22 23### Libraries 24 25Most external libraries are embedded in the project in the [/lib](/lib) folder. To keep its footprint low and make cross-platform builds simple, Fluent Bit attempts keep its dependency graph small. 26 27The external library you are mostly likely to interact with is [msgpack](https://github.com/msgpack/msgpack-c). 28 29For crypto, Fluent Bit uses [mbedtls](https://github.com/ARMmbed/mbedtls). 30 31#### Memory Management 32 33When you write Fluent Bit code, you will use Fluent Bit's versions of the standard C functions for working with memory: 34- [`flb_malloc()`](include/fluent-bit/flb_mem.h) - equivalent to `malloc`, allocates memory. 35- [`flb_calloc()`](include/fluent-bit/flb_mem.h) - equivalent to `calloc`, allocates memory and initializes it to zero. 36- [`flb_realloc()`](include/fluent-bit/flb_mem.h) - equivalent to `realloc`. 37- [`flb_free()`](include/fluent-bit/flb_mem.h) - equivalent to `free`, releases allocated memory. 38 39Note that many types have a specialized create and destroy function. For example, 40[`flb_sds_create()` and `flb_sds_destroy()`](include/fluent-bit/flb_sds.h) (more about this in the next section). 41 42#### Strings 43 44Fluent Bit has a stripped down version of the popular [SDS](https://github.com/antirez/sds) string library. See [flb_sds.h](include/fluent-bit/flb_sds.h) for the API. 45 46In general, you should use SDS strings in any string processing code. SDS strings are fully compatible with any C function that accepts a null-terminated sequence of characters; to understand how they work, see the [explanation on Github](https://github.com/antirez/sds#how-sds-strings-work). 47 48#### HTTP Client 49 50Fluent Bit has its own network connection library. The key types and functions are defined in the following header files: 51- [flb_upstream.h](include/fluent-bit/flb_upstream.h) 52- [flb_http_client.h](include/fluent-bit/flb_http_client.h) 53- [flb_io.h](include/fluent-bit/flb_io.h) 54 55The following code demonstrates making an HTTP request in Fluent Bit: 56 57```c 58#include <fluent-bit/flb_upstream.h> 59#include <fluent-bit/flb_io.h> 60#include <fluent-bit/flb_http_client.h> 61#include <fluent-bit/flb_info.h> 62#include <fluent-bit/flb_config.h> 63 64#define HOST "127.0.0.1" 65#define PORT 80 66 67static flb_sds_t make_request(struct flb_config *config) 68{ 69 struct flb_upstream *upstream; 70 struct flb_http_client *client; 71 size_t b_sent; 72 int ret; 73 struct flb_upstream_conn *u_conn; 74 flb_sds_t resp; 75 76 /* Create an 'upstream' context */ 77 upstream = flb_upstream_create(config, HOST, PORT, FLB_IO_TCP, NULL); 78 if (!upstream) { 79 flb_error("[example] connection initialization error"); 80 return -1; 81 } 82 83 /* Retrieve a TCP connection from the 'upstream' context */ 84 u_conn = flb_upstream_conn_get(upstream); 85 if (!u_conn) { 86 flb_error("[example] connection initialization error"); 87 flb_upstream_destroy(upstream); 88 return -1; 89 } 90 91 /* Create HTTP Client request/context */ 92 client = flb_http_client(u_conn, 93 FLB_HTTP_GET, metadata_path, 94 NULL, 0, 95 FLB_FILTER_AWS_IMDS_V2_HOST, 80, 96 NULL, 0); 97 98 if (!client) { 99 flb_error("[example] count not create http client"); 100 flb_upstream_conn_release(u_conn); 101 flb_upstream_destroy(upstream); 102 return -1; 103 } 104 105 /* Perform the HTTP request */ 106 ret = flb_http_do(client, &b_sent) 107 108 /* Validate return status and HTTP status if set */ 109 if (ret != 0 || client->resp.status != 200) { 110 if (client->resp.payload_size > 0) { 111 flb_debug("[example] Request failed and returned: \n%s", 112 client->resp.payload); 113 } 114 flb_http_client_destroy(client); 115 flb_upstream_conn_release(u_conn); 116 flb_upstream_destroy(upstream); 117 return -1; 118 } 119 120 /* Copy payload response to an output SDS buffer */ 121 data = flb_sds_create_len(client->resp.payload, 122 client->resp.payload_size); 123 124 flb_http_client_destroy(client); 125 flb_upstream_conn_release(u_conn); 126 flb_upstream_destroy(upstream); 127 128 return resp; 129} 130``` 131 132An `flb_upstream` structure represents a host/endpoint that you want to call. Normally, you'd store this structure somewhere so that it can be re-used. An `flb_upstream_conn` represents a connection to that host for a single HTTP request. The connection structure should not be used for more than one request. 133 134#### Linked Lists 135 136Fluent Bit contains a library for constructing linked lists- [mk_list](lib/monkey/include/monkey/mk_core/mk_list.h). The type stores data as a circular linked list. 137 138The [`mk_list.h`](lib/monkey/include/monkey/mk_core/mk_list.h) header file contains several macros and functions for use with the lists. The example below shows how to create a list, iterate through it, and delete an element. 139 140```c 141#include <monkey/mk_core/mk_list.h> 142#include <fluent-bit/flb_info.h> 143 144struct item { 145 char some_data; 146 147 struct mk_list _head; 148}; 149 150static int example() 151{ 152 struct mk_list *tmp; 153 struct mk_list *head; 154 struct mk_list items; 155 int i; 156 int len; 157 char characters[] = "abcdefghijk"; 158 struct item *an_item; 159 160 len = strlen(characters); 161 162 /* construct a list */ 163 mk_list_init(&items); 164 165 for (i = 0; i < len; i++) { 166 an_item = flb_malloc(sizeof(struct item)); 167 if (!an_item) { 168 flb_errno(); 169 return -1; 170 } 171 an_item->some_data = characters[i]; 172 mk_list_add(&an_item->_head, &items); 173 } 174 175 /* iterate through the list */ 176 flb_info("Iterating through list"); 177 mk_list_foreach_safe(head, tmp, &items) { 178 an_item = mk_list_entry(head, struct item, _head); 179 flb_info("list item data value: %c", an_item->some_data); 180 } 181 182 /* remove an item */ 183 mk_list_foreach_safe(head, tmp, &items) { 184 an_item = mk_list_entry(head, struct item, _head); 185 if (an_item->some_data == 'b') { 186 mk_list_del(&an_item->_head); 187 flb_free(an_item); 188 } 189 } 190} 191``` 192 193#### Message Pack 194 195Fluent Bit uses [msgpack](https://msgpack.org/index.html) to internally store data. If you write code for Fluent Bit, it is almost certain that you will interact with msgpack. 196 197Fluent Bit embeds the [msgpack-c](https://github.com/msgpack/msgpack-c) library. The example below shows manipulating message pack to add a new key-value pair to a record. In Fluent Bit, the [filter_record_modifier](plugins/filter_record_modifier) plugin adds or deletes keys from records. See its code for more. 198 199```c 200#define A_NEW_KEY "key" 201#define A_NEW_KEY_LEN 3 202#define A_NEW_VALUE "value" 203#define A_NEW_VALUE_LEN 5 204 205static int cb_filter(const void *data, size_t bytes, 206 const char *tag, int tag_len, 207 void **out_buf, size_t *out_size, 208 struct flb_filter_instance *f_ins, 209 void *context, 210 struct flb_config *config) 211{ 212 (void) f_ins; 213 (void) config; 214 size_t off = 0; 215 int i = 0; 216 int ret; 217 struct flb_time tm; 218 int total_records; 219 int new_keys = 1; 220 msgpack_sbuffer tmp_sbuf; 221 msgpack_packer tmp_pck; 222 msgpack_unpacked result; 223 msgpack_object *obj; 224 msgpack_object_kv *kv; 225 226 /* Create temporary msgpack buffer */ 227 msgpack_sbuffer_init(&tmp_sbuf); 228 msgpack_packer_init(&tmp_pck, &tmp_sbuf, msgpack_sbuffer_write); 229 230 /* Iterate over each item */ 231 msgpack_unpacked_init(&result); 232 while (msgpack_unpack_next(&result, data, bytes, &off) == MSGPACK_UNPACK_SUCCESS) { 233 /* 234 * Each record is a msgpack array [timestamp, map] of the 235 * timestamp and record map. We 'unpack' each record, and then re-pack 236 * it with the new fields added. 237 */ 238 239 if (result.data.type != MSGPACK_OBJECT_ARRAY) { 240 continue; 241 } 242 243 /* unpack the array of [timestamp, map] */ 244 flb_time_pop_from_msgpack(&tm, &result, &obj); 245 246 /* obj should now be the record map */ 247 if (obj->type != MSGPACK_OBJECT_MAP) { 248 continue; 249 } 250 251 /* re-pack the array into a new buffer */ 252 msgpack_pack_array(&tmp_pck, 2); 253 flb_time_append_to_msgpack(&tm, &tmp_pck, 0); 254 255 /* new record map size is old size + the new keys we will add */ 256 total_records = obj->via.map.size + new_keys; 257 msgpack_pack_map(&tmp_pck, total_records); 258 259 /* iterate through the old record map and add it to the new buffer */ 260 kv = obj->via.map.ptr; 261 for(i=0; i < obj->via.map.size; i++) { 262 msgpack_pack_object(&tmp_pck, (kv+i)->key); 263 msgpack_pack_object(&tmp_pck, (kv+i)->val); 264 } 265 266 /* append new keys */ 267 msgpack_pack_str(&tmp_pck, A_NEW_KEY_LEN); 268 msgpack_pack_str_body(&tmp_pck, A_NEW_KEY, A_NEW_KEY_LEN); 269 msgpack_pack_str(&tmp_pck, A_NEW_VALUE_LEN); 270 msgpack_pack_str_body(&tmp_pck, A_NEW_VALUE, A_NEW_VALUE_LEN); 271 272 } 273 msgpack_unpacked_destroy(&result); 274 275 /* link new buffers */ 276 *out_buf = tmp_sbuf.data; 277 *out_size = tmp_sbuf.size; 278 return FLB_FILTER_MODIFIED; 279``` 280 281Please also check out the message pack examples on the [msgpack-c GitHub repo](https://github.com/msgpack/msgpack-c). 282 283### Concurrency 284 285Fluent Bit uses ["coroutines"](https://en.wikipedia.org/wiki/Coroutine); a concurrent programming model in which subroutines can be paused and resumed. Co-routines are cooperative routines- instead of blocking, they cooperatively pass execution between each other. Coroutines are implemented as part of Fluent Bit's core network IO libraries. When a blocking network IO operation is made (for example, waiting for a response on a socket), a routine will cooperatively yield (pause itself) and pass execution to Fluent Bit engine, which will schedule (activate) other routines. Once the blocking IO operation is complete, the sleeping coroutine will be scheduled again (resumed). This model allows Fluent Bit to achieve performance benefits without the headaches that often come from having multiple active threads. 286 287This Fluent Bit engine consists of an event loop that is built upon [github.com/monkey/monkey](https://github.com/monkey/monkey). The monkey project is a server and library designed for low resource usage. It was primarily implemented by Eduardo Silva, who also created Fluent Bit. 288 289#### Coroutine Code: How does it work? 290 291To understand how this works, let's walkthrough an example in the code. 292 293The elasticsearch plugin makes an HTTP request to an elasticsearch cluster, when the following [line of code runs](https://github.com/fluent/fluent-bit/blob/1.3/plugins/out_es/es.c#L581): 294```c 295ret = flb_http_do(c, &b_sent); 296``` 297 298This calls the http request function, in [`flb_http_client.c`, which makes a TCP write call](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_http_client.c#L840): 299```c 300ret = flb_io_net_write(c->u_conn, 301 c->body_buf, c->body_len, 302 &bytes_body); 303``` 304 305That activates code in Fluent Bit's core TCP library, which is where the coroutine magic happens. This code is in [flb_io.c](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_io.c#L241). After opening a socket, the code inserts an item on the event loop: 306```c 307ret = mk_event_add(u->evl, 308 u_conn->fd, 309 FLB_ENGINE_EV_THREAD, 310 MK_EVENT_WRITE, &u_conn->event); 311``` 312 313This instructs the event loop to watch our socket's file descriptor. Then, [a few lines below, we yield back to the engine thread](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_io.c#L304): 314```c 315/* 316 * Return the control to the parent caller, we need to wait for 317 * the event loop to get back to us. 318 */ 319flb_thread_yield(th, FLB_FALSE); 320``` 321 322Remember, only one thread is active at a time. If the current coroutine did not yield back to engine, it would monopolize execution until the socket IO operation was complete. Since IO operations may take a long time, we can increase performance by allowing another routine to perform work. 323 324The core routine in Fluent Bit is the engine in `flb_engine.c`. Here we can find the [code that will resume the elasticsearch plugin](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_engine.c#L553) once it's IO operation is complete: 325```c 326if (event->type == FLB_ENGINE_EV_THREAD) { 327 struct flb_upstream_conn *u_conn; 328 struct flb_thread *th; 329 330 /* 331 * Check if we have some co-routine associated to this event, 332 * if so, resume the co-routine 333 */ 334 u_conn = (struct flb_upstream_conn *) event; 335 th = u_conn->thread; 336 flb_trace("[engine] resuming thread=%p", th); 337 flb_thread_resume(th); 338} 339``` 340 341This will return execution to the code right after the [flb_thread_yield](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_io.c#L304) call in the IO library. 342 343#### Practical Advice: How coroutines will affect your code 344 345##### Filter Plugins 346 347Filter plugins do not support coroutines, consequently you must disable async mode if your filter makes an HTTP request: 348```c 349/* Remove async flag from upstream */ 350upstream->flags &= ~(FLB_IO_ASYNC); 351``` 352 353##### Output plugins 354 355Output plugins use coroutines. Plugins have a context structure which is available in all calls and can be used to store state. In general, you can write code without ever considering concurrency. This is because only one coroutine is active at a time. Thus, synchronization primitives like mutex locks or semaphores are not needed. 356 357There are some cases where you need to consider concurrency; consider the following code (this is fluent bit c pseudo-code, not a full example): 358 359```c 360/* output plugin flush method for sending records */ 361static void cb_my_plugin_flush(...) 362{ 363 /* context structure that allows the plugin to store state */ 364 struct flb_my_plugin *ctx = out_context; 365 ... 366 /* write something to context */ 367 ctx->flag = somevalue; 368 369 /* make an async http call */ 370 ret = flb_http_do(c, &b_sent); 371 372 /* 373 * do something with the context flag; the value of flag is indeterminate 374 * because we just made an async call. 375 */ 376 somecall(ctx->flag); 377} 378``` 379 380When the http call is made, the current coroutine may be paused and another can be scheduled. That other coroutine may also call `cb_my_plugin_flush`. If that happens, the value of the `flag` on the context may be changed. This could potentially lead to a race condition when the first coroutine resumes. Consequently, you must be extremely careful when storing state on the context. In general, context values should be set when a plugin is initialized, and then should only be read from afterwards. 381 382Remember, if needed, you can ensure that an HTTP call is made synchronously by modifying your flb_upstream: 383 384```c 385/* Remove async flag from upstream */ 386upstream->flags &= ~(FLB_IO_ASYNC); 387``` 388 389This can be re-enabled at any time: 390 391```c 392/* re-enable async for future calls */ 393upstream->flags |= FLB_IO_ASYNC; 394``` 395 396 397### Plugin API 398 399Each plugin is a shared object which is [loaded into Fluent Bit](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_plugin.c#L70) using dlopen and dlsym. 400 401#### Input 402 403The input plugin structure is defined in [flb_input.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_input.h#L62). There are a number of functions which a plugin can implement, most only implement `cb_init`, `cb_collect`, and `cb_exit`. 404 405The [`"dummy"` input plugin](plugins/in_dummy) very simple and is an excellent example to review to understand more. 406 407#### Filter 408 409The structure for filter plugins is defined in [flb_filter.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_filter.h#L44). Each plugin must implement `cb_init`, `cb_filter`, and `cb_exit`. 410 411The [filter_record_modifier](plugins/filter_record_modifier) is a good example of a filter plugin. 412 413Note that filter plugins can not asynchronously make HTTP requests. If your plugin needs to make a request, add the following code when you initialize your `flb_upstream`: 414 415```c 416/* Remove async flag from upstream */ 417upstream->flags &= ~(FLB_IO_ASYNC); 418``` 419 420#### Output 421 422Output plugins are defined in [flb_output.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_output.h#L57). Each plugin must implement `cb_init`, `cb_flush`, and `cb_exit`. 423 424The [stdout plugin](plugins/out_stdout) is very simple; review its code to understand how output plugins work. 425 426#### Config Maps 427 428Config maps are an improvement to the previous Fluent Bit API that was used by plugins to read configuration values. The new config maps feature warns the user if there is an unknown configuration key and reduces risk of bad configuration due to typos or deprecated property names. They will also allow dynamic configuration reloading to be implemented in the future. 429 430There are various types of supported configuration types. Full list available [here](https://github.com/fluent/fluent-bit/blob/v1.4.2/include/fluent-bit/flb_config_map.h#L29). The most used ones are: 431 432| Type | Description | 433| -----------------------|:---------------------:| 434| FLB_CONFIG_MAP_INT | Represents integer data type | 435| FLB_CONFIG_MAP_BOOL | Represents boolean data type | 436| FLB_CONFIG_MAP_DOUBLE | Represents a double | 437| FLB_CONFIG_MAP_SIZE | Provides size_type as an integer datatype large enough to represent any possible string size. | 438| FLB_CONFIG_MAP_STR | Represents string data type | 439| FLB_CONFIG_MAP_CLIST | Comma separated list of strings | 440| FLB_CONFIG_MAP_SLIST | Empty space separated list of strings | 441 442A config map expects certain public fields at registration. 443 444| Public Fields | Description | 445| --------------|:---------------------| 446| Type | This field is the data type of the property that we are writing to the config map. If the property is of type `int` we use `FLB_CONFIG_MAP_INT`, if `string` `FLB_CONFIG_MAP_STR` etc. | 447| Name | This field is the name of the configuration property. For example for the property flush count we use `flush_count`| 448| Default Value | This field allows the user to set the default value of the property. For example, for a property of type `FLB_CONFIG_MAP_BOOL` (boolean), the default value may be false. Then we have to give `false` as default value. If there is no default value, `NULL` is given.| 449| Flags | This field allows the user to set option flags. For example, it specifies in certain cases if multiple entries are allowed. | 450| Set Property | This field decides if the property needs to be written to plugin context or just validated. If the property needs to be written to the plugin context, the value of this field needs to `FLB_TRUE` or else the value will be `FLB_FALSE`.| 451| Offset | This field represents the member offset. It is 0 if the property is not written to the plugin context and if the property is being written to the plugin context it is ```offsetof(struct name_of_plugin_structure, name_of_property)```. The macro offsetof() returns the offset of the field *member* from the start of the structure type.| 452| Description | This field is so that the user can give a short description of the property. It is `NULL` if no description is needed or given. | 453 454For example for [stdout](https://github.com/fluent/fluent-bit/blob/v1.4.2/plugins/out_stdout/stdout.c#L158) plugin the config map is something like: 455 456```c 457/* Configuration properties map */ 458static struct flb_config_map config_map[] = { 459 { 460 FLB_CONFIG_MAP_STR, "format", NULL, 461 0, FLB_FALSE, 0, 462 "Specifies the data format to be printed. Supported formats are msgpack json, json_lines and json_stream." 463 }, 464 { 465 FLB_CONFIG_MAP_STR, "json_date_format", NULL, 466 0, FLB_FALSE, 0, 467 "Specifies the name of the date field in output." 468 }, 469 { 470 FLB_CONFIG_MAP_STR, "json_date_key", "date", 471 0, FLB_TRUE, offsetof(struct flb_stdout, json_date_key), 472 "Specifies the format of the date. Supported formats are double, iso8601 and epoch." 473 }, 474 475 /* EOF */ 476 {0} 477}; 478 479/* Plugin registration */ 480struct flb_output_plugin out_stdout_plugin = { 481 .name = "stdout", 482 .description = "Prints events to STDOUT", 483 .cb_init = cb_stdout_init, 484 .cb_flush = cb_stdout_flush, 485 .cb_exit = cb_stdout_exit, 486 .flags = 0, 487 .config_map = config_map 488}; 489 490``` 491In the above code snippet, the property *format* is of type string which supports formats like json, msgpack etc. It has default value NULL(in which case it uses msgpack), no flags, and it is being only validated by the config map and hence set_property field is `FLB_FALSE` with member offset 0. No description is written for *format* property at present. 492Similarly, for the property *json_date_key*, type is string, default value is date, and it is being written to context so the set_property field is `FLB_TRUE` with a member offset. Again, no description is written for it. 493 494 495Upon initilization the engine loads the config map like [this](https://github.com/fluent/fluent-bit/blob/v1.4.2/plugins/out_stdout/stdout.c#L48): 496 497```c 498 ret = flb_output_config_map_set(ins, (void *) ctx); 499``` 500 501[flb_output_config_map_set](https://github.com/fluent/fluent-bit/blob/v1.4.2/include/fluent-bit/flb_output.h#L510) returns [flb_config_map_set](https://github.com/fluent/fluent-bit/blob/v1.4.2/src/flb_config_map.c#L513) which is a function used by plugins that needs to populate their context structure with the configuration properties already mapped. 502 503Some points to keep in mind while migrating an existing plugin to a config map interface: 504- All memory allocations and releases of properties on exit are handled by the config map interface. 505- The config map does not parse host and port properties since these properties are handled automatically for plugins that perform network operations. 506- Some plugins might also have an empty config_map. This is so that it would show an error when someone tried to use a non-existent parameter. 507 508### Testing 509 510During development, you can build Fluent Bit as follows: 511 512``` 513cd build 514cmake -DFLB_DEV=On ../ 515make 516``` 517Note that Fluent Bit uses Cmake 3 and on some systems you may need to invoke it as `cmake3`. 518 519To enable the unit tests run: 520``` 521cmake -DFLB_DEV=On -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On ../ 522make 523``` 524 525Internal tests are for the internal libraries of Fluent Bit. Runtime tests are for the plugins. 526 527You can run the unit tests with `make test`, however, this is inconvenient in practice. Each test file will create an executable in the `build/bin` directory which you can run directly. For example, if you want to run the SDS tests, you can invoke them as follows: 528 529``` 530$ ./bin/flb-it-sds 531Test sds_usage... [ OK ] 532Test sds_printf... [ OK ] 533SUCCESS: All unit tests have passed. 534``` 535 536#### Valgrind 537 538[Valgrind](https://valgrind.org/) is a tool that will help you detect and diagnose memory issues in your code. It will check for memory leaks and invalid memory accesses. 539 540To use it while developing, invoke it before Fluent Bit: 541 542``` 543valgrind ./bin/fluent-bit {args for fluent bit} 544``` 545 546Valgrind becomes especially powerful when you run it on your unit tests. We recommend writing unit tests that cover a large fraction of code paths in your contribution. You can then check your code for memory issues by invoking the test binaries with Valgrind: 547 548``` 549$ valgrind ./bin/flb-rt-your-test 550``` 551 552This will allow you to check for memory issues in code paths (ex error cases) which are hard to trigger through manual testing. 553 554### Need more help? 555 556The best way to learn how Fluent Bit code works is to read it. If you need help understanding the code, reach out to the community, or open a PR with changes that are a work in progress. 557