1Plugins register for Blosc users
2=============================================================================
3
4Blosc has a tradition of supporting different filters and codecs for compressing data,
5and it was up to the user to choose one or another depending on his needs.
6However, it is clear that there will always be scenarios where a more richer variety
7of them could be useful.  So the Blosc Development Team has set new goals:
8
91) Implement a way for users to locally register filters and codecs so that they can use
10   them in their setup at will.
11
122) Setup a central registry so that *other* users can make use of these filters and codecs
13   without intefering with other ones that have been created by other users.
14
15As a bonus, those codecs and filters accepted in the central registry and meeting the quality standards
16defined in these guidelines will be distributed *inside* the C-Blosc2 library,
17allowing a much easier way for others to use them: install C-Blosc2 library and you are all set.
18Of course, to achieve such a status, plugins will require a careful testing process described below.
19
20
21Plugin types
22--------------
23
24The plugins that are registered in the repository can be codecs or filters.
25
26A **codec** is a program able to compress and decompress a digital data stream
27with the objective of reduce dataset size to enable a faster transmission
28of data.
29Some of the codecs used by Blosc are e.g. *BLOSCLZ*, *LZ4* and *ZSTANDARD*.
30
31A **filter** is a program that reorders the data without
32changing its size, so that the initial and final size are equal.
33A filter consists of encoder and decoder. Filter encoder is applyed before
34using the codec compressor (or codec encoder) in order to make data easier to compress
35and filter decoder is used after codec decompressor (or codec decoder) to recover
36the original data arrangement.
37Some filters actually used by Blosc are e.g. *SHUFFLE*, which rearranges data
38based on the typesize, or *TRUNC*, which zeroes mantissa bits so as to reduce
39the precision of (floating point) data, and hence, increase the compression ratio.
40
41Here it is an example on how the compression process goes:
42
43
44    --------------------   filter encoder  -------------------   codec encoder   -------
45    |        src        |   ----------->  |        tmp        |   ---------->   | c_src |
46    --------------------                   -------------------                   -------
47
48And the decompression process:
49
50    --------   codec decoder    -------------------   filter decoder  -------------------
51    | c_src |    ----------->  |        tmp        |   ---------->   |        src        |
52    --------                    -------------------                   -------------------
53
54Moreover, during the pipeline process you can use even 6 different
55filters ordered as you prefer.
56
57
58Blosc global registered plugins vs user registered plugins
59----------------------------------------------------------
60
61**Blosc global registered plugins** are official Blosc plugins that have passed through a selection process
62and have been recognised by the Blosc Development Team. These plugins are available for
63everybody in the C-Blosc2 GitHub repository and users can install them anytime.
64
65**User registered plugins** are plugins that users register locally and they can use them
66in the same way as in the examples `urcodecs.c` and `urfilters.c`.
67
68If you only want to use a plugin on your own devices you can just register it as a user registered
69plugin with an ID between *BLOSC2_USER_REGISTERED_FILTERS_START* and *BLOSC2_USER_REGISTERED_FILTERS_STOP*.
70Otherwise, if you think that your plugin could be useful for the community you can apply for
71registering it as an official Blosc plugin following the next steps.
72
73
74Requirements for registering plugins
75------------------------------------
76
77For users wanting to register a new codec or filter, there are some requirements
78that their code must satisfy:
79
80- First, the plugin code must be **developed in C**, have a relatively small footprint
81  and meet decent quality code standards.
82
83- Second, users must develop a test suite which prove that the plugin works correctly.
84
85Finally, even if these requirements are completely satisfied, it is not
86guaranteed that the plugin will be useful or contribute something
87different than the existing ones, so the Blosc development team has the final
88say and will decide if a plugin is to be accepted or not.
89
90
91Steps
92-----
93
941. First, tests must be provided and be passing.
95
96   **It is completely mandatory and necessary to add these lines to `main()` in each test to make plugins machinery work:**
97   - `blosc_init()` at the beginning
98   - `blosc_destroy()` in the end
99
100
1012. Then, the user must make a fork of the C-Blosc2 Github repository,
102   adding a new folder within the plugin sources to the path `plugins/codecs` or
103   `plugins/filters` depending on the plugin type.
104
1053. Furthermore, a text file named `README.rst` must be provided where it is explained:
106
107   * The plugin motivation, why and for what purpose was the plugin created.
108
109   * How to use the plugin.
110
111   * What does the plugin do and how it works.
112
113   * The advantages and disadvantages of the plugin compared to the rest.
114
1154. To register a plugin the user must choose a plugin ID between *BLOSC2_GLOBAL_REGISTERED_FILTERS_START* and *BLOSC2_GLOBAL_REGISTERED_FILTERS_STOP* and
116   write it at `include/blosc2/codecs-registry.h`
117   or `include/blosc2/filters-registry.h` depending on the plugin type. Then, you have to edit `include/blosc2/codecs-registry.c`or
118
119   `include/blosc2/filters-registry.c` in the next way:
120
121   At the top it must be added `#include "plugin_folder/plugin_header.h"`,
122
123   and into the register function you must follow the same steps that were done for the existing plugins.
124
1255. Finally, the Blosc development team will carry out the evaluation process
126   (probably via a votation process, with the BDFL having the last say in case of the team is undecided)
127   so as to decide whether the plugin is useful and hence, candidate to be integrated into the C-Blosc2
128   source code distribution.  In case of a negative decision, this will be properly stated.
129
130
131Examples
132--------
133
134In the `plugins/` directory there can be found different examples of codecs and filters
135available as plugins that can be used in the compression process, and that
136can be used as an example on how to implement plugins that can make into C-Blosc2.
137Some of these examples are `ndlz`, `ndcell` or `ndmean`.
138
139
140Thanks
141------
142
143We would like to express our gratitude to the NumFOCUS Foundation so as to provide the funds to implement this functionality.
144
145