1Plugins register for Blosc users 2============================================================================= 3 4Blosc has a tradition of supporting different filters and codecs for compressing data, 5and it was up to the user to choose one or another depending on his needs. 6However, it is clear that there will always be scenarios where a more richer variety 7of them could be useful. So the Blosc Development Team has set new goals: 8 91) Implement a way for users to locally register filters and codecs so that they can use 10 them in their setup at will. 11 122) Setup a central registry so that *other* users can make use of these filters and codecs 13 without intefering with other ones that have been created by other users. 14 15As a bonus, those codecs and filters accepted in the central registry and meeting the quality standards 16defined in these guidelines will be distributed *inside* the C-Blosc2 library, 17allowing a much easier way for others to use them: install C-Blosc2 library and you are all set. 18Of course, to achieve such a status, plugins will require a careful testing process described below. 19 20 21Plugin types 22-------------- 23 24The plugins that are registered in the repository can be codecs or filters. 25 26A **codec** is a program able to compress and decompress a digital data stream 27with the objective of reduce dataset size to enable a faster transmission 28of data. 29Some of the codecs used by Blosc are e.g. *BLOSCLZ*, *LZ4* and *ZSTANDARD*. 30 31A **filter** is a program that reorders the data without 32changing its size, so that the initial and final size are equal. 33A filter consists of encoder and decoder. Filter encoder is applyed before 34using the codec compressor (or codec encoder) in order to make data easier to compress 35and filter decoder is used after codec decompressor (or codec decoder) to recover 36the original data arrangement. 37Some filters actually used by Blosc are e.g. *SHUFFLE*, which rearranges data 38based on the typesize, or *TRUNC*, which zeroes mantissa bits so as to reduce 39the precision of (floating point) data, and hence, increase the compression ratio. 40 41Here it is an example on how the compression process goes: 42 43 44 -------------------- filter encoder ------------------- codec encoder ------- 45 | src | -----------> | tmp | ----------> | c_src | 46 -------------------- ------------------- ------- 47 48And the decompression process: 49 50 -------- codec decoder ------------------- filter decoder ------------------- 51 | c_src | -----------> | tmp | ----------> | src | 52 -------- ------------------- ------------------- 53 54Moreover, during the pipeline process you can use even 6 different 55filters ordered as you prefer. 56 57 58Blosc global registered plugins vs user registered plugins 59---------------------------------------------------------- 60 61**Blosc global registered plugins** are official Blosc plugins that have passed through a selection process 62and have been recognised by the Blosc Development Team. These plugins are available for 63everybody in the C-Blosc2 GitHub repository and users can install them anytime. 64 65**User registered plugins** are plugins that users register locally and they can use them 66in the same way as in the examples `urcodecs.c` and `urfilters.c`. 67 68If you only want to use a plugin on your own devices you can just register it as a user registered 69plugin with an ID between *BLOSC2_USER_REGISTERED_FILTERS_START* and *BLOSC2_USER_REGISTERED_FILTERS_STOP*. 70Otherwise, if you think that your plugin could be useful for the community you can apply for 71registering it as an official Blosc plugin following the next steps. 72 73 74Requirements for registering plugins 75------------------------------------ 76 77For users wanting to register a new codec or filter, there are some requirements 78that their code must satisfy: 79 80- First, the plugin code must be **developed in C**, have a relatively small footprint 81 and meet decent quality code standards. 82 83- Second, users must develop a test suite which prove that the plugin works correctly. 84 85Finally, even if these requirements are completely satisfied, it is not 86guaranteed that the plugin will be useful or contribute something 87different than the existing ones, so the Blosc development team has the final 88say and will decide if a plugin is to be accepted or not. 89 90 91Steps 92----- 93 941. First, tests must be provided and be passing. 95 96 **It is completely mandatory and necessary to add these lines to `main()` in each test to make plugins machinery work:** 97 - `blosc_init()` at the beginning 98 - `blosc_destroy()` in the end 99 100 1012. Then, the user must make a fork of the C-Blosc2 Github repository, 102 adding a new folder within the plugin sources to the path `plugins/codecs` or 103 `plugins/filters` depending on the plugin type. 104 1053. Furthermore, a text file named `README.rst` must be provided where it is explained: 106 107 * The plugin motivation, why and for what purpose was the plugin created. 108 109 * How to use the plugin. 110 111 * What does the plugin do and how it works. 112 113 * The advantages and disadvantages of the plugin compared to the rest. 114 1154. To register a plugin the user must choose a plugin ID between *BLOSC2_GLOBAL_REGISTERED_FILTERS_START* and *BLOSC2_GLOBAL_REGISTERED_FILTERS_STOP* and 116 write it at `include/blosc2/codecs-registry.h` 117 or `include/blosc2/filters-registry.h` depending on the plugin type. Then, you have to edit `include/blosc2/codecs-registry.c`or 118 119 `include/blosc2/filters-registry.c` in the next way: 120 121 At the top it must be added `#include "plugin_folder/plugin_header.h"`, 122 123 and into the register function you must follow the same steps that were done for the existing plugins. 124 1255. Finally, the Blosc development team will carry out the evaluation process 126 (probably via a votation process, with the BDFL having the last say in case of the team is undecided) 127 so as to decide whether the plugin is useful and hence, candidate to be integrated into the C-Blosc2 128 source code distribution. In case of a negative decision, this will be properly stated. 129 130 131Examples 132-------- 133 134In the `plugins/` directory there can be found different examples of codecs and filters 135available as plugins that can be used in the compression process, and that 136can be used as an example on how to implement plugins that can make into C-Blosc2. 137Some of these examples are `ndlz`, `ndcell` or `ndmean`. 138 139 140Thanks 141------ 142 143We would like to express our gratitude to the NumFOCUS Foundation so as to provide the funds to implement this functionality. 144 145