1.. _rfc-51:
2
3=======================================================================================
4RFC 51: RasterIO() improvements : resampling and progress callback
5=======================================================================================
6
7Author: Even Rouault
8
9Contact: even dot rouault at spatialys dot com
10
11Status: Adopted, implemented in GDAL 2.0
12
13Summary
14-------
15
16This RFC aims at extending the RasterIO() API to allow specifying a
17resampling algorithm when doing requests involving subsampling or
18oversampling. A progress callback can also be specified to be notified
19of progression and allow the user to interrupt the operation.
20
21Core changes
22------------
23
24Addition of GDALRasterIOExtraArg structure
25~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
26
27A new structure GDALRasterIOExtraArg is added to contain the new
28options.
29
30::
31
32   /** Structure to pass extra arguments to RasterIO() method
33     * @since GDAL 2.0
34     */
35   typedef struct
36   {
37       /*! Version of structure (to allow future extensions of the structure) */
38       int                    nVersion;
39
40       /*! Resampling algorithm */
41       GDALRIOResampleAlg     eResampleAlg;
42
43       /*! Progress callback */
44       GDALProgressFunc       pfnProgress;
45       /*! Progress callback user data */
46       void                  *pProgressData;
47
48       /*! Indicate if dfXOff, dfYOff, dfXSize and dfYSize are set.
49           Mostly reserved from the VRT driver to communicate a more precise
50           source window. Must be such that dfXOff - nXOff < 1.0 and
51           dfYOff - nYOff < 1.0 and nXSize - dfXSize < 1.0 and nYSize - dfYSize < 1.0 */
52       int                    bFloatingPointWindowValidity;
53       /*! Pixel offset to the top left corner. Only valid if bFloatingPointWindowValidity = TRUE */
54       double                 dfXOff;
55       /*! Line offset to the top left corner. Only valid if bFloatingPointWindowValidity = TRUE */
56       double                 dfYOff;
57       /*! Width in pixels of the area of interest. Only valid if bFloatingPointWindowValidity = TRUE */
58       double                 dfXSize;
59       /*! Height in pixels of the area of interest. Only valid if bFloatingPointWindowValidity = TRUE */
60       double                 dfYSize;
61   } GDALRasterIOExtraArg;
62
63   #define RASTERIO_EXTRA_ARG_CURRENT_VERSION  1
64
65   /** Macro to initialize an instance of GDALRasterIOExtraArg structure.
66     * @since GDAL 2.0
67     */
68   #define INIT_RASTERIO_EXTRA_ARG(s)  \
69       do { (s).nVersion = RASTERIO_EXTRA_ARG_CURRENT_VERSION; \
70            (s).eResampleAlg = GRIORA_NearestNeighbour; \
71            (s).pfnProgress = NULL; \
72            (s).pProgressData = NULL; \
73            (s).bFloatingPointWindowValidity = FALSE; } while(0)
74
75There are several reasons to prefer a structure rather than new
76parameters to the RasterIO() methods :
77
78-  code readability (GDALDataset::IRasterIO() has already 14
79   parameters...)
80-  allow future extensions without changing the prototype in all drivers
81-  to a lesser extent, efficiency: it is common for RasterIO() calls to
82   be chained between generic/specific and/or dataset/rasterband
83   implementations. Passing just the pointer is more efficient.
84
85The structure is versioned. In the future if further options are added,
86the new members will be added at the end of the structure and the
87version number will be incremented. Code in GDAL core&drivers can check
88the version number to determine which options are available.
89
90Addition of GDALRIOResampleAlg structure
91~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
92
93The following resampling algorithms are available :
94
95::
96
97   /** RasterIO() resampling method.
98     * @since GDAL 2.0
99     */
100   typedef enum
101   {
102       /*! Nearest neighbour */                            GRIORA_NearestNeighbour = 0,
103       /*! Bilinear (2x2 kernel) */                        GRIORA_Bilinear = 1,
104       /*! Cubic Convolution Approximation (4x4 kernel) */ GRIORA_Cubic = 2,
105       /*! Cubic B-Spline Approximation (4x4 kernel) */    GRIORA_CubicSpline = 3,
106       /*! Lanczos windowed sinc interpolation (6x6 kernel) */ GRIORA_Lanczos = 4,
107       /*! Average */                                      GRIORA_Average = 5,
108       /*! Mode (selects the value which appears most often of all the sampled points) */
109                                                           GRIORA_Mode = 6,
110       /*! Gauss blurring */                               GRIORA_Gauss = 7
111   } GDALRIOResampleAlg;
112
113Those new resampling methods can be used by the
114GDALRasterBand::IRasterIO() default implementation when the size of the
115buffer (nBufXSize x nBufYSize) is different from the size of the area of
116interest (nXSize x nYSize). The code heavily relies on the algorithms
117used for overview computation, with adjustments to be also able to deal
118with oversampling. Bilinear, CubicSpline and Lanczos are now available
119in overview computation as well, and rely on the generic infrastructure
120for convolution computation introduced lately for improved cubic
121overviews. Some algorithms are not available on raster bands with color
122palette. A warning will be emitted if an attempt of doing so is done,
123and nearest neighbour will be used as a fallback.
124
125The GDAL_RASTERIO_RESAMPLING configuration option can be set as an
126alternate way of specifying the resampling algorithm. Mainly useful for
127tests with applications that do not yet use the new API.
128
129Currently, the new resampling methods are only available for GF_Read
130operations. The use case for GF_Write operations isn't obvious, but
131could be added without API changes if needed.
132
133C++ changes
134~~~~~~~~~~~
135
136GDALDataset and GDALRasterBand (non virtual) RasterIO() and (virtual)
137IRasterIO() methods have a new final argument psExtraArg of type
138GDALRasterIOExtraArg*. This extra argument defaults to NULL for code
139using GDAL, but is required for all in-tree code, so as to avoid that
140in-tree code forgets to forwards psExtraArg it might have returned from
141a caller.
142
143GDALDataset::RasterIO() and GDALRasterBand::RasterIO() can accept a NULL
144pointer for that argument in which case they will instantiate a default
145GDALRasterIOExtraArg structure to be passed to IRasterIO(). Any other
146code that calls IRasterIO() directly (a few IReadBlock()
147implementations) should make sure of doing so, so that IRasterIO() can
148assume that its psExtraArg is not NULL.
149
150As a provision to be able to deal with very large requests with buffers
151larger than several gigabytes, the nPixelSpace, nLineSpace and
152nBandSpace parameters have been promoted from the int datatype to the
153new GSpacing datatype, which is an alias of a signed 64 bit integer.
154
155GDALRasterBand::IRasterIO() and GDALDataset::BlockBasedRasterIO() now
156use the progress callback when available.
157
158C API changes
159~~~~~~~~~~~~~
160
161Only additions :
162
163::
164
165   CPLErr CPL_DLL CPL_STDCALL GDALDatasetRasterIOEx(
166       GDALDatasetH hDS, GDALRWFlag eRWFlag,
167       int nDSXOff, int nDSYOff, int nDSXSize, int nDSYSize,
168       void * pBuffer, int nBXSize, int nBYSize, GDALDataType eBDataType,
169       int nBandCount, int *panBandCount,
170       GSpacing nPixelSpace, GSpacing nLineSpace, GSpacing nBandSpace,
171       GDALRasterIOExtraArg* psExtraArg);
172
173   CPLErr CPL_DLL CPL_STDCALL
174   GDALRasterIOEx( GDALRasterBandH hRBand, GDALRWFlag eRWFlag,
175                   int nDSXOff, int nDSYOff, int nDSXSize, int nDSYSize,
176                   void * pBuffer, int nBXSize, int nBYSize,GDALDataType eBDataType,
177                   GSpacing nPixelSpace, GSpacing nLineSpace,
178                   GDALRasterIOExtraArg* psExtraArg );
179
180Those are the same as the existing functions with a final
181GDALRasterIOExtraArg\* psExtraArg argument, and the spacing parameters
182promoted to GSpacing.
183
184Changes in drivers
185------------------
186
187-  All in-tree drivers that implemented or used RasterIO have been
188   edited to accept the GDALRasterIOExtraArg\* psExtraArg parameter, and
189   forward it when needed. Those who had a custom RasterIO()
190   implementation now use the progress callback when available.
191-  VRT: the and elements can accept a 'resampling' attribute. The VRT
192   driver will also set the dfXOff, dfYOff, dfXSize and dfYSize fields
193   of GDALRasterIOExtraArg\* to have source sub-pixel accuracy, so that
194   GDALRasterBand::IRasterIO() leads to consistent results when
195   operating on a small area of interest or the whole raster. If that
196   was not done, chunking done in GDALDatasetCopyWholeRaster() or other
197   algorithms could lead to repeated lines due to integer rounding
198   issues.
199
200Changes in utilities
201--------------------
202
203-  gdal_translate: accept a -r parameter to specify the resampling
204   algorithm. Defaults to NEAR. Can be set to bilinear, cubic,
205   cubicspline, lanczos, average or mode. (Under the hood, this sets the
206   new resampling property at the VRT source level.)
207-  gdaladdo: -r parameter now accepts bilinear, cubicspline and lanczos
208   as additional algorithms to the existing ones.
209
210Changes in SWIG bindings
211------------------------
212
213-  For Python and Perl bindings: Band.ReadRaster(), Dataset.ReadRaster()
214   now accept optional resample_alg, callback and callback_data
215   arguments. (untested for Perl, but the existing tests pass)
216-  For Python bindings, Band.ReadAsArray() and Dataset.ReadAsArray() now
217   accept optional resample_alg, callback and callback_data arguments.
218
219Compatibility
220-------------
221
222-  C API/ABI preserved.
223
224-  C++ users of the GDALRasterBand::RasterIO() and
225   GDALDataset::RasterIO() API do not need to change their code, since
226   the new GDALRasterIOExtraArg\* psExtraArg argument is optional for
227   out-of-tree code.
228
229-  Out-of-tree drivers that implement IRasterIO() must be changed to
230   accept the new GDALRasterIOExtraArg\* psExtraArg argument. Note:
231   failing to do so will be undetected at compile time (due to how C++
232   virtual method overloading work).
233
234Both issues will be mentioned in MIGRATION_GUIDE.TXT
235
236Documentation
237-------------
238
239All new methods are documented.
240
241Testing
242-------
243
244The various aspects of this RFC are tested in the Python bindings:
245
246-  use of the new options of Band.ReadRaster(), Dataset.ReadRaster(),
247   Band.ReadAsArray() and Dataset.ReadAsArray().
248-  resampling algorithms in subsampling and oversampling RasterIO()
249   requests.
250-  "-r" option of gdal_translate
251
252Implementation
253--------------
254
255Implementation will be done by Even Rouault
256(`Spatialys <http://spatialys.com>`__), and sponsored by `R3
257GIS <http://r3-gis.com>`__.
258
259The proposed implementation lies in the "rasterio" branch of the
260`https://github.com/rouault/gdal2/tree/rasterio <https://github.com/rouault/gdal2/tree/rasterio>`__
261repository.
262
263The list of changes :
264`https://github.com/rouault/gdal2/compare/rasterio <https://github.com/rouault/gdal2/compare/rasterio>`__
265
266Voting history
267--------------
268
269+1 from FrankW, JukkaR, HowardB, DanielM, TamasS and EvenR
270