1.. _rfc-75:
2
3================================================================================
4RFC 75: Multidimensional arrays
5================================================================================
6
7============== ============================
8Author:        Even Rouault
9Contact:       even.rouault @ spatialys.com
10Started:       2019-May-24
11Last updated:  2019-Jul-22
12Status:        Implemented in GDAL 3.1
13============== ============================
14
15Summary
16-------
17
18This document describes the addition of read/write support for multidimensional
19arrays, in particular of dimension 3 or above, in GDAL core and a few select drivers.
20
21Motivation
22----------
23
24Multidimensional arrays (also known as hypercubes) are a way of mоdelling
25spatio-temporal (time series of
262D raster) or spatio-vertical-temporal (2D + Z dimension + time dimension) data which
27are becoming increasingly more available. GDAL current raster model is however strongly
282D oriented. A number of drivers, such as netCDF, HDF4, HDF5, work around that
29limitation by using raster bands or subdatasets to expose muliple 2D slices of what
30is intrinsically a N>2 Multidimensional dataset. It is desirable to have a
31proper API, and driver support, to be able to expose those multidimensional
32arrays as such, and be able to perform slice and trim operations on them.
33
34That topic had already been discussed in the past, in particular in this
35`mailing list thread <https://lists.osgeo.org/pipermail/gdal-dev/2017-October/047472.html>`_
36
37Proposed changes
38----------------
39
40A lot of existing GDAL raster API are strongly 2D oriented. Rather than upgrading
41all those API, and driver code, to be ready for N arbitrary dimensions, which would
42be a enormous effort for the benefit of only a small fraction of drivers, we propose
43to add a new dedicated API to support multidimensional arrays. We also want
44to support hierarchical structure of data as found in the
45`HDF5 format and data model <https://portal.opengeospatial.org/files/81716>`_.
46This model can encompass the needs of other formats/drivers that have multidimensional
47capabilities such as HDF4, netCDF, GRIB, WCS.
48Therefore the proposed API will be strongly inspired by the API of the HDF5 library itself.
49
50Data model
51~~~~~~~~~~
52The data model is described in:
53https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/multidim_raster_data_model.rst
54
55C++ API
56~~~~~~~
57
58New classes and methods will be added.
59See https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal_priv.h#L1715
60
61A new driver capability will be added for drivers supporting multidimensional
62rasters:
63
64::
65
66    #define GDAL_DCAP_MULTIDIM_RASTER     "DCAP_MULTIDIM_RASTER"
67
68
69A new open flag, ``GDAL_OF_MULTIDIM_RASTER``, for :cpp:func:`GDALOpenEx`
70will be added. When this is specified, drivers supporting multidimensional
71raster will return a root GDALGroup. Otherwise their current traditional 2D
72mode will still be used.
73
74New creation options metadata items are added to documents multidimensional dataset,
75group, dimension, array and attribute creation options.
76
77.. code-block:: c++
78
79    /** XML snippet with multidimensional dataset creation options.
80    * @since GDAL 3.1
81    */
82    #define GDAL_DMD_MULTIDIM_DATASET_CREATIONOPTIONLIST "DMD_MULTIDIM_DATASET_CREATIONOPTIONLIST"
83
84    /** XML snippet with multidimensional group creation options.
85    * @since GDAL 3.1
86    */
87    #define GDAL_DMD_MULTIDIM_GROUP_CREATIONOPTIONLIST "DMD_MULTIDIM_GROUP_CREATIONOPTIONLIST"
88
89    /** XML snippet with multidimensional dimension creation options.
90    * @since GDAL 3.1
91    */
92    #define GDAL_DMD_MULTIDIM_DIMENSION_CREATIONOPTIONLIST "DMD_MULTIDIM_DIMENSION_CREATIONOPTIONLIST"
93
94    /** XML snippet with multidimensional array creation options.
95    * @since GDAL 3.1
96    */
97    #define GDAL_DMD_MULTIDIM_ARRAY_CREATIONOPTIONLIST "DMD_MULTIDIM_ARRAY_CREATIONOPTIONLIST"
98
99    /** XML snippet with multidimensional attribute creation options.
100    * @since GDAL 3.1
101    */
102    #define GDAL_DMD_MULTIDIM_ATTRIBUTE_CREATIONOPTIONLIST "DMD_MULTIDIM_ATTRIBUTE_CREATIONOPTIONLIST"
103
104Examples with the netCDF driver:
105
106.. code-block:: xml
107
108    <MultiDimDatasetCreationOptionList>
109    <Option name="FORMAT" type="string-select" default="NC4">
110        <Value>NC</Value>
111        <Value>NC2</Value>
112        <Value>NC4</Value>
113        <Value>NC4C</Value>
114    </Option>
115    <Option name="CONVENTIONS" type="string" default="CF-1.6" description="Value of the Conventions attribute" />
116    </MultiDimDatasetCreationOptionList>
117
118
119    <MultiDimDimensionCreationOptionList>
120    <Option name="UNLIMITED" type="boolean" description="Whether the dimension should be unlimited" default="false" />
121    </MultiDimDimensionCreationOptionList>
122
123
124    <MultiDimArrayCreationOptionList>
125    <Option name="BLOCKSIZE" type="int" description="Block size in pixels" />
126    <Option name="COMPRESS" type="string-select" default="NONE">
127        <Value>NONE</Value>
128        <Value>DEFLATE</Value>
129    </Option>
130    <Option name="ZLEVEL" type="int" description="DEFLATE compression level 1-9" default="1" />
131    <Option name="NC_TYPE" type="string-select" default="netCDF data type">
132        <Value>AUTO</Value>
133        <Value>NC_BYTE</Value>
134        <Value>NC_INT64</Value>
135        <Value>NC_UINT64</Value>
136    </Option>
137    </MultiDimArrayCreationOptionList>
138
139
140    <MultiDimAttributeCreationOptionList>
141    <Option name="NC_TYPE" type="string-select" default="netCDF data type">
142        <Value>AUTO</Value>
143        <Value>NC_BYTE</Value>
144        <Value>NC_CHAR</Value>
145        <Value>NC_INT64</Value>
146        <Value>NC_UINT64</Value>
147    </Option>
148    </MultiDimAttributeCreationOptionList>
149
150
151C API
152~~~~~
153
154All C++ methods are mapped to the C API.
155See https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal.h#L1397
156
157Driver changes
158~~~~~~~~~~~~~~
159
160- The MEM driver will implement read and write support.
161- The VRT driver will allow extraction of 2D slices from multidimensional
162  drivers to 2D/classic drivers, as well as multidimensional->multidimensional
163  slicing/trimming
164- The netCDF driver will implement read and write support.
165- The HDF4 and HDF5 drivers will implement read support.
166- The GRIB driver will implement read support (exposing X,Y,Time arrays for GRIB
167  messages only differing by timestamp)
168
169New Utilities
170~~~~~~~~~~~~~
171
172- A new gdalmdiminfo utility is added to report the hierarchical structure and content.
173  Its output format is JSON. See https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdiminfo.rst
174  for its documentation.
175
176- A new gdalmdimtranslate utility is added to convert multidimensional raster between
177  different formats, and/or can perform selective conversion of specific arrays
178  and groups, and/or subsetting operations. It can also do extraction of 2D slices
179  from multidimensional drivers to 2D/classic drivers.
180  See https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdimtranslate.rst
181  for its documentation.
182
183SWIG binding changes
184~~~~~~~~~~~~~~~~~~~~
185
186The C API is mapped to the SWIG bindings. The scope is complete for the
187Python bindings. Other languages would need to add missing typemaps, but this
188is not in the scope of the work of this RFC.
189For Python bindings, NumPy integration is done.
190
191Limitations
192-----------
193
194This is intended to be a preliminary work on that topic. While the aim is for it
195to be be usable for the defined scope, it will probably require future
196enhancements to fill functional and/or performance gaps.
197
198- No block cache mechanism (not sure this is needed)
199- No sub-pixel requests, or non-nearest subsampling
200- Upgrade of WCS driver or other drivers with potential multidimensional
201  capabilities are not part of this RFC.
202- SWIG bindings: full scope only for Python bindings.
203
204Backward compatibility
205----------------------
206
207No backward incompatibility. Only API and utility additions.
208
209Documentation
210-------------
211
212- Data model: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/multidim_raster_data_model.rst
213- API tutorial: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/tutorials/multidimensional_api_tut.rst
214- gdalmdiminfo: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdiminfo.rst
215- gdalmdimtranslate: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdimtranslate.rst
216- VRT driver: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/drivers/raster/vrt_multidimensional.rst
217
218Testing
219-------
220
221The gdalautotest suite is extended to test the modified drivers and the new
222utilities.
223
224Implementation
225--------------
226
227The implementation will be done by Even Rouault.
228A preliminary implementation is available at
229https://github.com/OSGeo/gdal/pull/1704
230
231Voting history
232--------------
233
234+1 from HowardB, NormanB and EvenR
235