1.. _rfc-24:
2
3================================================================================
4RFC 24: GDAL Progressive Data Support
5================================================================================
6
7Author: Norman Barker, Frank Warmerdam
8
9Contact: nbarker@ittvis.com, warmerdam@pobox.com
10
11Status: Adopted
12
13Summary
14-------
15
16Provide an interface for asynchronous/streaming data access in GDAL. The
17initial implementation is for JPIP, but should be generic enough to
18apply to other streaming / progressive approaches. Background on the
19JPIP (Kakadu) implementation can be found in [wiki:rfc24_jpipkak].
20
21Interfaces
22----------
23
24GDALAsyncReader
25~~~~~~~~~~~~~~~
26
27This new class is intended to represent an active asynchronous raster
28imagery request. The request includes information on a source window on
29the dataset, a target buffer size (implies level of decimation or
30replication), the buffer type, buffer interleaving, data buffer and
31bands being requested. Essentially the same sort of information that is
32passed in a GDALDataset::!RasterIO() request.
33
34The GetNextUpdatedRegion() method can be used to wait for an update to
35the imagery buffer, and to find out what area was updated. The
36LockBuffer() and UnlockBuffer() methods can be used to temporarily
37disable updates to the buffer while application code accesses the
38buffer.
39
40While an implementation of the simple accessors is provided as part of
41the class, it is intended that the class be subclassed as part of
42implementation of a particular driver, and custom implementations of
43GetNextUpdatedRegion(), LockBuffer() and UnlockBuffer() provided.
44
45{{{ class CPL_DLL GDALAsyncReader { protected: GDALDataset\* poDS; int
46nXOff; int nYOff; int nXSize; int nYSize; void \* pBuf; int nBufXSize;
47int nBufYSize; GDALDataType eBufType; int nBandCount; int\* panBandMap;
48int nPixelSpace; int nLineSpace; int nBandSpace; long nDataRead;
49
50public: GDALAsyncReader(GDALDataset\* poDS = NULL); virtual
51~GDALAsyncReader();
52
53::
54
55   GDALDataset* GetGDALDataset() {return poDS;}
56   int GetXOffset() {return nXOff;}
57   int GetYOffset() {return nYOff;}
58   int GetXSize() {return nXSize;}
59   int GetYSize() {return nYSize;}
60   void * GetBuffer() {return pBuf;}
61   int GetBufferXSize() {return nBufXSize;}
62   int GetBufferYSize() {return nBufYSize;}
63   GDALDataType GetBufferType() {return eBufType;}
64   int GetBandCount() {return nBandCount;}
65   int* GetBandMap() {return panBandMap;}
66   int GetPixelSpace() {return nPixelSpace;}
67   int GetLineSpace() {return nLineSpace;}
68   int GetBandSpace() {return nBandSpace;}
69
70   virtual GDALAsyncStatusType GetNextUpdatedRegion(double dfTimeout,
71                                                    int* pnBufXOff,
72                                                    int* pnBufYOff,
73                                                    int* pnBufXSize,
74                                                    int* pnBufXSize) = 0;
75
76   virtual int LockBuffer( double dfTimeout );
77   virtual void UnlockBuffer();
78
79   friend class GDALDataset;
80
81}; }}}
82
83GetNextUpdatedRegion()
84~~~~~~~~~~~~~~~~~~~~~~
85
86::
87
88   GDALAsyncStatusType
89   GDALAsyncRasterio::GetNextUpdatedRegion(int dfTimeout,
90                                           int* pnBufXOff, int* pnBufYOff,
91                                           int* pnBufXSize, int* pnBufXSize);
92
93   int dfTimeout;
94     The amount of time to wait for results measured  in seconds.  If this is
95     zero available work may be processed but no waiting for the arrival of more
96     imagery should be done.  A value of -1.0 means wait an infinite amount of
97     time for new data.  Processing available imagery may still take an
98     arbitrary amount of time.
99
100   int *pnBufXOff, *pnBufYOff, *pnBufXSize, *pnBufYSize;
101     The window of data updated within the async io imagery buffer is returned in
102     these variables. This information can be used to limit screen redraws or other
103     processing to the portion of the imagery that may have changed.
104
105The async return status list is as follows, and will be declared in
106gdal.h.
107
108::
109
110   typedef enum
111   {
112       GARIO_PENDING = 0,
113       GARIO_UPDATE = 1,
114       GARIO_ERROR = 2,
115       GARIO_COMPLETE = 3,
116       GARIO_TypeCount = 4
117   } GDALAsyncStatusType;
118
119The meaning as a return value is:
120
121-  GARIO_PENDING: No imagery was altered in the buffer, but there is
122   still activity pending, and the application should continue to call
123   GetNextUpdatedRegion() as time permits.
124-  GARIO_UPDATE: Some of the imagery has been updated, but there is
125   still activity pending.
126-  GARIO_ERROR: Something has gone wrong. The asynchronous request
127   should be ended.
128-  GARIO_COMPLETE: An update has occurred and there is no more pending
129   work on this request. The request should be ended and the buffer
130   used.
131
132GDALDataset
133~~~~~~~~~~~
134
135The GDALDataset class is extended with methods to create an asynchronous
136reader, and to cleanup the asynchronous reader. It is intended that
137these methods would be subclassed by drivers implementing asynchronous
138data access.
139
140::
141
142       virtual GDALAsyncReader*
143           BeginAsyncReader(int nXOff, int nYOff, int nXSize, int nYSize,
144                              void *pBuf, int nBufXSize, int nBufYSize,
145                              GDALDataType eBufType,
146                              int nBandCount, int* panBandMap,
147                              int nPixelSpace, int nLineSpace, int nBandSpace,
148                              char **papszOptions);
149       virtual void EndAsyncReader(GDALAsyncReader *);
150
151It is expected that as part of gdal/gcore a default !GDALAsyncReader
152implementation will be provided that just uses GDALDataset::!RasterIO()
153to perform the request as a single blocking request. However, this
154default implementation will ensure that applications can use the
155asynchronous interface without worrying whether a particular format will
156actually operate asynchronously.
157
158GDALDriver
159~~~~~~~~~~
160
161In order to provide a hint to applications whether particular formats
162support asynchronous IO, we will add a new metadata item on the
163GDALDriver of implementing formats. The metadata item will be
164"DCAP_ASYNCIO" (macro GDAL_DCAP_ASYNCIO) and will have the value "YES"
165if asynchronous IO is available.
166
167Implementing drivers will do something like this in their driver setup
168code:
169
170::
171
172      poDriver->SetMetadataItem( GDAL_DCAP_ASYNCIO, "YES" );
173
174GDALRasterBand
175~~~~~~~~~~~~~~
176
177There are no changes to the GDALRasterBand interface for asynchronous
178raster IO. Asynchronous IO requests can only be made at the dataset
179level, not the band.
180
181C API
182-----
183
184The following C API wrappers for the C++ classes and methods will be
185added. Note that at this time there is no intention to provide C
186wrappers for all the GDALAsyncReader accessors since the provided
187information is already available in the application from the call
188launching the async io.
189
190::
191
192   typedef void *GDALAsyncReaderH;
193
194   GDALAsyncStatusType CPL_DLL CPL_STDCALL
195   GDALGetNextUpdatedRegion(GDALAsyncReaderH hARIO, double dfTimeout,
196                            int* pnXBufOff, int* pnYBufOff,
197                            int* pnXBufSize, int* pnYBufSize );
198   int CPL_DLL CPL_STDCALL GDALLockBuffer(GDALAsyncReaderH hARIO,double dfTimeout);
199   void CPL_DLL CPL_STDCALL GDALUnlockBuffer(GDALAsyncReaderH hARIO);
200
201   GDALAsyncReaderH CPL_DLL CPL_STDCALL
202   GDALBeginAsyncReader(GDALDatasetH hDS, int nXOff, int nYOff,
203                          int nXSize, int nYSize,
204                          void *pBuf, int nBufXSize, int nBufYSize,
205                          GDALDataType eBufType,
206                          int nBandCount, int* panBandMap,
207                          int nPixelSpace, int nLineSpace, int nBandSpace,
208                          char **papszOptions);
209   void  CPL_DLL CPL_STDCALL
210   GDALEndAsyncReader(GDALDatasetH hDS, GDALAsyncReaderH hAsynchRasterIOH);
211
212SWIG
213----
214
215It is intended that all the above functions in the C API will be wrapped
216for SWIG.
217
218Driver Implementations
219----------------------
220
221A full implementation of the Asynchronous API will be provided as the
222JPIPKAK driver - a JPIP protocol implementation using the Kakadu
223library.
224
225At this time, no other implementations are planned.
226
227Testing
228-------
229
230Some testing of the asynchronous api against normal drivers will be
231added in the test suite, as well as testing of the JPIPKAK driver in
232asynchronous and conventional data access methods.
233
234Also, a new commandline program, gdalasyncread, is implemented which
235provides a mechanism to test the async API from the commandline. It
236takes a subset of the gdal_translate commandline options.
237
238::
239
240   Usage: gdalasyncread [--help-general]
241          [-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/
242                CInt16/CInt32/CFloat32/CFloat64}]
243          [-of format] [-b band]
244          [-outsize xsize[%] ysize[%]]
245          [-srcwin xoff yoff xsize ysize]
246          [-co "NAME=VALUE"]* [-ao "NAME=VALUE"]
247          src_dataset dst_dataset
248
249