1.. _rfc-75: 2 3================================================================================ 4RFC 75: Multidimensional arrays 5================================================================================ 6 7============== ============================ 8Author: Even Rouault 9Contact: even.rouault @ spatialys.com 10Started: 2019-May-24 11Last updated: 2019-Jul-22 12Status: Implemented in GDAL 3.1 13============== ============================ 14 15Summary 16------- 17 18This document describes the addition of read/write support for multidimensional 19arrays, in particular of dimension 3 or above, in GDAL core and a few select drivers. 20 21Motivation 22---------- 23 24Multidimensional arrays (also known as hypercubes) are a way of mоdelling 25spatio-temporal (time series of 262D raster) or spatio-vertical-temporal (2D + Z dimension + time dimension) data which 27are becoming increasingly more available. GDAL current raster model is however strongly 282D oriented. A number of drivers, such as netCDF, HDF4, HDF5, work around that 29limitation by using raster bands or subdatasets to expose muliple 2D slices of what 30is intrinsically a N>2 Multidimensional dataset. It is desirable to have a 31proper API, and driver support, to be able to expose those multidimensional 32arrays as such, and be able to perform slice and trim operations on them. 33 34That topic had already been discussed in the past, in particular in this 35`mailing list thread <https://lists.osgeo.org/pipermail/gdal-dev/2017-October/047472.html>`_ 36 37Proposed changes 38---------------- 39 40A lot of existing GDAL raster API are strongly 2D oriented. Rather than upgrading 41all those API, and driver code, to be ready for N arbitrary dimensions, which would 42be a enormous effort for the benefit of only a small fraction of drivers, we propose 43to add a new dedicated API to support multidimensional arrays. We also want 44to support hierarchical structure of data as found in the 45`HDF5 format and data model <https://portal.opengeospatial.org/files/81716>`_. 46This model can encompass the needs of other formats/drivers that have multidimensional 47capabilities such as HDF4, netCDF, GRIB, WCS. 48Therefore the proposed API will be strongly inspired by the API of the HDF5 library itself. 49 50Data model 51~~~~~~~~~~ 52The data model is described in: 53https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/multidim_raster_data_model.rst 54 55C++ API 56~~~~~~~ 57 58New classes and methods will be added. 59See https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal_priv.h#L1715 60 61A new driver capability will be added for drivers supporting multidimensional 62rasters: 63 64:: 65 66 #define GDAL_DCAP_MULTIDIM_RASTER "DCAP_MULTIDIM_RASTER" 67 68 69A new open flag, ``GDAL_OF_MULTIDIM_RASTER``, for :cpp:func:`GDALOpenEx` 70will be added. When this is specified, drivers supporting multidimensional 71raster will return a root GDALGroup. Otherwise their current traditional 2D 72mode will still be used. 73 74New creation options metadata items are added to documents multidimensional dataset, 75group, dimension, array and attribute creation options. 76 77.. code-block:: c++ 78 79 /** XML snippet with multidimensional dataset creation options. 80 * @since GDAL 3.1 81 */ 82 #define GDAL_DMD_MULTIDIM_DATASET_CREATIONOPTIONLIST "DMD_MULTIDIM_DATASET_CREATIONOPTIONLIST" 83 84 /** XML snippet with multidimensional group creation options. 85 * @since GDAL 3.1 86 */ 87 #define GDAL_DMD_MULTIDIM_GROUP_CREATIONOPTIONLIST "DMD_MULTIDIM_GROUP_CREATIONOPTIONLIST" 88 89 /** XML snippet with multidimensional dimension creation options. 90 * @since GDAL 3.1 91 */ 92 #define GDAL_DMD_MULTIDIM_DIMENSION_CREATIONOPTIONLIST "DMD_MULTIDIM_DIMENSION_CREATIONOPTIONLIST" 93 94 /** XML snippet with multidimensional array creation options. 95 * @since GDAL 3.1 96 */ 97 #define GDAL_DMD_MULTIDIM_ARRAY_CREATIONOPTIONLIST "DMD_MULTIDIM_ARRAY_CREATIONOPTIONLIST" 98 99 /** XML snippet with multidimensional attribute creation options. 100 * @since GDAL 3.1 101 */ 102 #define GDAL_DMD_MULTIDIM_ATTRIBUTE_CREATIONOPTIONLIST "DMD_MULTIDIM_ATTRIBUTE_CREATIONOPTIONLIST" 103 104Examples with the netCDF driver: 105 106.. code-block:: xml 107 108 <MultiDimDatasetCreationOptionList> 109 <Option name="FORMAT" type="string-select" default="NC4"> 110 <Value>NC</Value> 111 <Value>NC2</Value> 112 <Value>NC4</Value> 113 <Value>NC4C</Value> 114 </Option> 115 <Option name="CONVENTIONS" type="string" default="CF-1.6" description="Value of the Conventions attribute" /> 116 </MultiDimDatasetCreationOptionList> 117 118 119 <MultiDimDimensionCreationOptionList> 120 <Option name="UNLIMITED" type="boolean" description="Whether the dimension should be unlimited" default="false" /> 121 </MultiDimDimensionCreationOptionList> 122 123 124 <MultiDimArrayCreationOptionList> 125 <Option name="BLOCKSIZE" type="int" description="Block size in pixels" /> 126 <Option name="COMPRESS" type="string-select" default="NONE"> 127 <Value>NONE</Value> 128 <Value>DEFLATE</Value> 129 </Option> 130 <Option name="ZLEVEL" type="int" description="DEFLATE compression level 1-9" default="1" /> 131 <Option name="NC_TYPE" type="string-select" default="netCDF data type"> 132 <Value>AUTO</Value> 133 <Value>NC_BYTE</Value> 134 <Value>NC_INT64</Value> 135 <Value>NC_UINT64</Value> 136 </Option> 137 </MultiDimArrayCreationOptionList> 138 139 140 <MultiDimAttributeCreationOptionList> 141 <Option name="NC_TYPE" type="string-select" default="netCDF data type"> 142 <Value>AUTO</Value> 143 <Value>NC_BYTE</Value> 144 <Value>NC_CHAR</Value> 145 <Value>NC_INT64</Value> 146 <Value>NC_UINT64</Value> 147 </Option> 148 </MultiDimAttributeCreationOptionList> 149 150 151C API 152~~~~~ 153 154All C++ methods are mapped to the C API. 155See https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal.h#L1397 156 157Driver changes 158~~~~~~~~~~~~~~ 159 160- The MEM driver will implement read and write support. 161- The VRT driver will allow extraction of 2D slices from multidimensional 162 drivers to 2D/classic drivers, as well as multidimensional->multidimensional 163 slicing/trimming 164- The netCDF driver will implement read and write support. 165- The HDF4 and HDF5 drivers will implement read support. 166- The GRIB driver will implement read support (exposing X,Y,Time arrays for GRIB 167 messages only differing by timestamp) 168 169New Utilities 170~~~~~~~~~~~~~ 171 172- A new gdalmdiminfo utility is added to report the hierarchical structure and content. 173 Its output format is JSON. See https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdiminfo.rst 174 for its documentation. 175 176- A new gdalmdimtranslate utility is added to convert multidimensional raster between 177 different formats, and/or can perform selective conversion of specific arrays 178 and groups, and/or subsetting operations. It can also do extraction of 2D slices 179 from multidimensional drivers to 2D/classic drivers. 180 See https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdimtranslate.rst 181 for its documentation. 182 183SWIG binding changes 184~~~~~~~~~~~~~~~~~~~~ 185 186The C API is mapped to the SWIG bindings. The scope is complete for the 187Python bindings. Other languages would need to add missing typemaps, but this 188is not in the scope of the work of this RFC. 189For Python bindings, NumPy integration is done. 190 191Limitations 192----------- 193 194This is intended to be a preliminary work on that topic. While the aim is for it 195to be be usable for the defined scope, it will probably require future 196enhancements to fill functional and/or performance gaps. 197 198- No block cache mechanism (not sure this is needed) 199- No sub-pixel requests, or non-nearest subsampling 200- Upgrade of WCS driver or other drivers with potential multidimensional 201 capabilities are not part of this RFC. 202- SWIG bindings: full scope only for Python bindings. 203 204Backward compatibility 205---------------------- 206 207No backward incompatibility. Only API and utility additions. 208 209Documentation 210------------- 211 212- Data model: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/multidim_raster_data_model.rst 213- API tutorial: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/tutorials/multidimensional_api_tut.rst 214- gdalmdiminfo: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdiminfo.rst 215- gdalmdimtranslate: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/programs/gdalmdimtranslate.rst 216- VRT driver: https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/drivers/raster/vrt_multidimensional.rst 217 218Testing 219------- 220 221The gdalautotest suite is extended to test the modified drivers and the new 222utilities. 223 224Implementation 225-------------- 226 227The implementation will be done by Even Rouault. 228A preliminary implementation is available at 229https://github.com/OSGeo/gdal/pull/1704 230 231Voting history 232-------------- 233 234+1 from HowardB, NormanB and EvenR 235