1.. _rfc-24: 2 3================================================================================ 4RFC 24: GDAL Progressive Data Support 5================================================================================ 6 7Author: Norman Barker, Frank Warmerdam 8 9Contact: nbarker@ittvis.com, warmerdam@pobox.com 10 11Status: Adopted 12 13Summary 14------- 15 16Provide an interface for asynchronous/streaming data access in GDAL. The 17initial implementation is for JPIP, but should be generic enough to 18apply to other streaming / progressive approaches. Background on the 19JPIP (Kakadu) implementation can be found in [wiki:rfc24_jpipkak]. 20 21Interfaces 22---------- 23 24GDALAsyncReader 25~~~~~~~~~~~~~~~ 26 27This new class is intended to represent an active asynchronous raster 28imagery request. The request includes information on a source window on 29the dataset, a target buffer size (implies level of decimation or 30replication), the buffer type, buffer interleaving, data buffer and 31bands being requested. Essentially the same sort of information that is 32passed in a GDALDataset::!RasterIO() request. 33 34The GetNextUpdatedRegion() method can be used to wait for an update to 35the imagery buffer, and to find out what area was updated. The 36LockBuffer() and UnlockBuffer() methods can be used to temporarily 37disable updates to the buffer while application code accesses the 38buffer. 39 40While an implementation of the simple accessors is provided as part of 41the class, it is intended that the class be subclassed as part of 42implementation of a particular driver, and custom implementations of 43GetNextUpdatedRegion(), LockBuffer() and UnlockBuffer() provided. 44 45{{{ class CPL_DLL GDALAsyncReader { protected: GDALDataset\* poDS; int 46nXOff; int nYOff; int nXSize; int nYSize; void \* pBuf; int nBufXSize; 47int nBufYSize; GDALDataType eBufType; int nBandCount; int\* panBandMap; 48int nPixelSpace; int nLineSpace; int nBandSpace; long nDataRead; 49 50public: GDALAsyncReader(GDALDataset\* poDS = NULL); virtual 51~GDALAsyncReader(); 52 53:: 54 55 GDALDataset* GetGDALDataset() {return poDS;} 56 int GetXOffset() {return nXOff;} 57 int GetYOffset() {return nYOff;} 58 int GetXSize() {return nXSize;} 59 int GetYSize() {return nYSize;} 60 void * GetBuffer() {return pBuf;} 61 int GetBufferXSize() {return nBufXSize;} 62 int GetBufferYSize() {return nBufYSize;} 63 GDALDataType GetBufferType() {return eBufType;} 64 int GetBandCount() {return nBandCount;} 65 int* GetBandMap() {return panBandMap;} 66 int GetPixelSpace() {return nPixelSpace;} 67 int GetLineSpace() {return nLineSpace;} 68 int GetBandSpace() {return nBandSpace;} 69 70 virtual GDALAsyncStatusType GetNextUpdatedRegion(double dfTimeout, 71 int* pnBufXOff, 72 int* pnBufYOff, 73 int* pnBufXSize, 74 int* pnBufXSize) = 0; 75 76 virtual int LockBuffer( double dfTimeout ); 77 virtual void UnlockBuffer(); 78 79 friend class GDALDataset; 80 81}; }}} 82 83GetNextUpdatedRegion() 84~~~~~~~~~~~~~~~~~~~~~~ 85 86:: 87 88 GDALAsyncStatusType 89 GDALAsyncRasterio::GetNextUpdatedRegion(int dfTimeout, 90 int* pnBufXOff, int* pnBufYOff, 91 int* pnBufXSize, int* pnBufXSize); 92 93 int dfTimeout; 94 The amount of time to wait for results measured in seconds. If this is 95 zero available work may be processed but no waiting for the arrival of more 96 imagery should be done. A value of -1.0 means wait an infinite amount of 97 time for new data. Processing available imagery may still take an 98 arbitrary amount of time. 99 100 int *pnBufXOff, *pnBufYOff, *pnBufXSize, *pnBufYSize; 101 The window of data updated within the async io imagery buffer is returned in 102 these variables. This information can be used to limit screen redraws or other 103 processing to the portion of the imagery that may have changed. 104 105The async return status list is as follows, and will be declared in 106gdal.h. 107 108:: 109 110 typedef enum 111 { 112 GARIO_PENDING = 0, 113 GARIO_UPDATE = 1, 114 GARIO_ERROR = 2, 115 GARIO_COMPLETE = 3, 116 GARIO_TypeCount = 4 117 } GDALAsyncStatusType; 118 119The meaning as a return value is: 120 121- GARIO_PENDING: No imagery was altered in the buffer, but there is 122 still activity pending, and the application should continue to call 123 GetNextUpdatedRegion() as time permits. 124- GARIO_UPDATE: Some of the imagery has been updated, but there is 125 still activity pending. 126- GARIO_ERROR: Something has gone wrong. The asynchronous request 127 should be ended. 128- GARIO_COMPLETE: An update has occurred and there is no more pending 129 work on this request. The request should be ended and the buffer 130 used. 131 132GDALDataset 133~~~~~~~~~~~ 134 135The GDALDataset class is extended with methods to create an asynchronous 136reader, and to cleanup the asynchronous reader. It is intended that 137these methods would be subclassed by drivers implementing asynchronous 138data access. 139 140:: 141 142 virtual GDALAsyncReader* 143 BeginAsyncReader(int nXOff, int nYOff, int nXSize, int nYSize, 144 void *pBuf, int nBufXSize, int nBufYSize, 145 GDALDataType eBufType, 146 int nBandCount, int* panBandMap, 147 int nPixelSpace, int nLineSpace, int nBandSpace, 148 char **papszOptions); 149 virtual void EndAsyncReader(GDALAsyncReader *); 150 151It is expected that as part of gdal/gcore a default !GDALAsyncReader 152implementation will be provided that just uses GDALDataset::!RasterIO() 153to perform the request as a single blocking request. However, this 154default implementation will ensure that applications can use the 155asynchronous interface without worrying whether a particular format will 156actually operate asynchronously. 157 158GDALDriver 159~~~~~~~~~~ 160 161In order to provide a hint to applications whether particular formats 162support asynchronous IO, we will add a new metadata item on the 163GDALDriver of implementing formats. The metadata item will be 164"DCAP_ASYNCIO" (macro GDAL_DCAP_ASYNCIO) and will have the value "YES" 165if asynchronous IO is available. 166 167Implementing drivers will do something like this in their driver setup 168code: 169 170:: 171 172 poDriver->SetMetadataItem( GDAL_DCAP_ASYNCIO, "YES" ); 173 174GDALRasterBand 175~~~~~~~~~~~~~~ 176 177There are no changes to the GDALRasterBand interface for asynchronous 178raster IO. Asynchronous IO requests can only be made at the dataset 179level, not the band. 180 181C API 182----- 183 184The following C API wrappers for the C++ classes and methods will be 185added. Note that at this time there is no intention to provide C 186wrappers for all the GDALAsyncReader accessors since the provided 187information is already available in the application from the call 188launching the async io. 189 190:: 191 192 typedef void *GDALAsyncReaderH; 193 194 GDALAsyncStatusType CPL_DLL CPL_STDCALL 195 GDALGetNextUpdatedRegion(GDALAsyncReaderH hARIO, double dfTimeout, 196 int* pnXBufOff, int* pnYBufOff, 197 int* pnXBufSize, int* pnYBufSize ); 198 int CPL_DLL CPL_STDCALL GDALLockBuffer(GDALAsyncReaderH hARIO,double dfTimeout); 199 void CPL_DLL CPL_STDCALL GDALUnlockBuffer(GDALAsyncReaderH hARIO); 200 201 GDALAsyncReaderH CPL_DLL CPL_STDCALL 202 GDALBeginAsyncReader(GDALDatasetH hDS, int nXOff, int nYOff, 203 int nXSize, int nYSize, 204 void *pBuf, int nBufXSize, int nBufYSize, 205 GDALDataType eBufType, 206 int nBandCount, int* panBandMap, 207 int nPixelSpace, int nLineSpace, int nBandSpace, 208 char **papszOptions); 209 void CPL_DLL CPL_STDCALL 210 GDALEndAsyncReader(GDALDatasetH hDS, GDALAsyncReaderH hAsynchRasterIOH); 211 212SWIG 213---- 214 215It is intended that all the above functions in the C API will be wrapped 216for SWIG. 217 218Driver Implementations 219---------------------- 220 221A full implementation of the Asynchronous API will be provided as the 222JPIPKAK driver - a JPIP protocol implementation using the Kakadu 223library. 224 225At this time, no other implementations are planned. 226 227Testing 228------- 229 230Some testing of the asynchronous api against normal drivers will be 231added in the test suite, as well as testing of the JPIPKAK driver in 232asynchronous and conventional data access methods. 233 234Also, a new commandline program, gdalasyncread, is implemented which 235provides a mechanism to test the async API from the commandline. It 236takes a subset of the gdal_translate commandline options. 237 238:: 239 240 Usage: gdalasyncread [--help-general] 241 [-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/ 242 CInt16/CInt32/CFloat32/CFloat64}] 243 [-of format] [-b band] 244 [-outsize xsize[%] ysize[%]] 245 [-srcwin xoff yoff xsize ysize] 246 [-co "NAME=VALUE"]* [-ao "NAME=VALUE"] 247 src_dataset dst_dataset 248 249