1.. _rfc-6:
2
3=======================================================================================
4RFC 6: Geometry and Feature Style as OGR Special Fields
5=======================================================================================
6
7Author: Tamas Szekeres
8
9Contact: szekerest@gmail.com
10
11Status: Adopted
12
13Summary
14-------
15
16This proposal addresses and issue have been discovered long ago, and OGR
17provides no equivalent solution so far.
18
19Some of the supported formats like Mapinfo.tab may contain multiple
20geometry types and style information. In order to handle this kind of
21data sources properly a support for selecting the layers by geometry
22type or by the style info would be highly required. For more details see
23the following MapServer related bugs later in this document.
24
25All of the proposed changes can be found at the tracking bug of this RFC
26referenced later in this document.
27
28Main concepts
29-------------
30
31The most reasonable way to support this feature is to extend the
32currently existing 'special field' approach to allow specifying more
33than one fields. Along with the already defined 'FID' field we will add
34the following ones:
35
36-  'OGR_GEOMETRY' containing the geometry type like 'POINT' or
37   'POLYGON'.
38-  'OGR_STYLE' containing the style string.
39-  'OGR_GEOM_WKT' containing the full WKT of the geometry.
40
41By providing the aforementioned fields one can make for example the
42following selections:
43
44-  select FID, OGR_GEOMETRY, OGR_STYLE, OGR_GEOM_WKT, \* from MyTable
45   where OGR_GEOMETRY='POINT' OR OGR_GEOMETRY='POLYGON'
46-  select FID, OGR_GEOMETRY, OGR_STYLE, OGR_GEOM_WKT, \* from MyTable
47   where OGR_STYLE LIKE '%BRUSH%'
48-  select FID, OGR_GEOMETRY, OGR_STYLE, OGR_GEOM_WKT, \* from MyTable
49   where OGR_GEOM_WKT LIKE 'POLYGON%'
50-  select distinct OGR_GEOMETRY from MyTable order by OGR_GEOMETRY desc
51
52Implementation
53--------------
54
55There are two distinct areas where this feature plays a role
56
57-  Feature query implemented at ogrfeaturequery.cpp
58
59-  SQL based selection implemented at ogr_gensql.cpp and
60   ogrdatasource.cpp
61
62To specify arbitrary number of special fields we will declare an array
63for the field names and types in ogrfeaturequery.cpp as
64
65::
66
67   char* SpecialFieldNames[SPECIAL_FIELD_COUNT]
68       = {"FID", "OGR_GEOMETRY", "OGR_STYLE", "OGR_GEOM_WKT"};
69   swq_field_type SpecialFieldTypes[SPECIAL_FIELD_COUNT]
70       = {SWQ_INTEGER, SWQ_STRING, SWQ_STRING, SWQ_STRING};
71
72So as to make this array accessible to the other files the followings
73will be added to ogr_p.h
74
75::
76
77   CPL_C_START
78   #include "ogr_swq.h"
79   CPL_C_END
80
81   #define SPF_FID 0
82   #define SPF_OGR_GEOMETRY 1
83   #define SPF_OGR_STYLE 2
84   #define SPF_OGR_GEOM_WKT 3
85   #define SPECIAL_FIELD_COUNT 4
86
87   extern char* SpecialFieldNames[SPECIAL_FIELD_COUNT];
88   extern swq_field_type SpecialFieldTypes[SPECIAL_FIELD_COUNT];
89
90In ogrfeature.cpp the field accessor functions (GetFieldAsString,
91GetFieldAsInteger, GetFieldAsDouble) will be modified providing the
92values of the special fields by the field index
93
94The following code will be added to the beginning of
95OGRFeature::GetFieldAsInteger:
96
97::
98
99   int iSpecialField = iField - poDefn->GetFieldCount();
100   if (iSpecialField >= 0)
101   {
102   // special field value accessors
103       switch (iSpecialField)
104       {
105       case SPF_FID:
106           return GetFID();
107       default:
108           return 0;
109       }
110   }
111
112The following code will be added to the beginning of
113OGRFeature::GetFieldAsDouble:
114
115::
116
117   int iSpecialField = iField - poDefn->GetFieldCount();
118   if (iSpecialField >= 0)
119   {
120   // special field value accessors
121       switch (iSpecialField)
122       {
123       case SPF_FID:
124           return GetFID();
125       default:
126           return 0.0;
127       }
128   }
129
130The following code will be added to the beginning of
131OGRFeature::GetFieldAsString:
132
133::
134
135   int iSpecialField = iField - poDefn->GetFieldCount();
136   if (iSpecialField >= 0)
137   {
138   // special field value accessors
139       switch (iSpecialField)
140       {
141       case SPF_FID:
142           sprintf( szTempBuffer, "%d", GetFID() );
143           return m_pszTmpFieldValue = CPLStrdup( szTempBuffer );
144       case SPF_OGR_GEOMETRY:
145           return poGeometry->getGeometryName();
146       case SPF_OGR_STYLE:
147           return GetStyleString();
148       case SPF_OGR_GEOM_WKT:
149           {
150               if (poGeometry->exportToWkt( &m_pszTmpFieldValue ) == OGRERR_NONE )
151                   return m_pszTmpFieldValue;
152               else
153                   return "";
154           }
155       default:
156           return "";
157       }
158   }
159
160The current implementation of OGRFeature::GetFieldAsString uses a static
161string to hold the const char\* return value that is highly avoidable
162and makes the code thread unsafe. In this regard the 'static char
163szTempBuffer[80]' will be changed to non static and a new member will be
164added to OGRFeature in ogrfeature.h as:
165
166::
167
168   char * m_pszTmpFieldValue;
169
170This member will be initialized to NULL at the constructor, and will be
171freed using CPLFree() at the destructor of OGRFeature.
172
173In OGRFeature::GetFieldAsString all of the occurrences of 'return
174szTempBuffer;' will be changed to 'return m_pszTmpFieldValue =
175CPLStrdup( szTempBuffer );'
176
177OGRFeature::GetFieldAsString is responsible to destroy the old value of
178m_pszTmpFieldValue at the beginning of the function:
179
180::
181
182   CPLFree(m_pszTmpFieldValue);
183   m_pszTmpFieldValue = NULL;
184
185In ogrfeaturequery.cpp we should change OGRFeatureQuery::Compile to add
186the special fields like:
187
188::
189
190   iField = 0;
191   while (iField < SPECIAL_FIELD_COUNT)
192   {
193       papszFieldNames[poDefn->GetFieldCount() + iField] = SpecialFieldNames[iField];
194       paeFieldTypes[poDefn->GetFieldCount() + iField] = SpecialFieldTypes[iField];
195       ++iField;
196   }
197
198In ogrfeaturequery.cpp OGRFeatureQueryEvaluator() should be modifyed
199according to the field specific actions like
200
201::
202
203   int iSpecialField = op->field_index - poFeature->GetDefnRef()->GetFieldCount();
204   if( iSpecialField >= 0 )
205   {
206       if ( iSpecialField < SPECIAL_FIELD_COUNT )
207       {
208           switch ( SpecialFieldTypes[iSpecialField] )
209           {
210           case SWQ_INTEGER:
211               sField.Integer = poFeature->GetFieldAsInteger( op->field_index );
212           case SWQ_STRING:
213               sField.String = (char*) poFeature->GetFieldAsString( op->field_index );
214           }
215       }
216       else
217       {
218           CPLDebug( "OGRFeatureQuery", "Illegal special field index.");
219           return FALSE;
220       }
221       psField = &sField;
222   }
223   else
224       psField = poFeature->GetRawFieldRef( op->field_index );
225
226In ogrfeaturequery.cpp OGRFeatureQuery::FieldCollector should be
227modifyed to add the field names like:
228
229::
230
231   if( op->field_index >= poTargetDefn->GetFieldCount()
232           && op->field_index < poTargetDefn->GetFieldCount() + SPECIAL_FIELD_COUNT)
233           pszFieldName = SpecialFieldNames[op->field_index];
234
235In ogrdatasource.cpp ExecuteSQL() will allocate the arrays according to
236the number of the special fields:
237
238::
239
240   sFieldList.names = (char **)
241           CPLMalloc( sizeof(char *) * (nFieldCount+SPECIAL_FIELD_COUNT) );
242   sFieldList.types = (swq_field_type *)
243           CPLMalloc( sizeof(swq_field_type) * (nFieldCount+SPECIAL_FIELD_COUNT) );
244   sFieldList.table_ids = (int *)
245           CPLMalloc( sizeof(int) * (nFieldCount+SPECIAL_FIELD_COUNT) );
246   sFieldList.ids = (int *)
247           CPLMalloc( sizeof(int) * (nFieldCount+SPECIAL_FIELD_COUNT) );
248
249And the fields will be added as
250
251::
252
253   for (iField = 0; iField < SPECIAL_FIELD_COUNT; iField++)
254   {
255       sFieldList.names[sFieldList.count] = SpecialFieldNames[iField];
256       sFieldList.types[sFieldList.count] = SpecialFieldTypes[iField];
257       sFieldList.table_ids[sFieldList.count] = 0;
258       sFieldList.ids[sFieldList.count] = nFIDIndex + iField;
259       sFieldList.count++;
260   }
261
262For supporting the SQL based queries we should also modify the
263constructor of OGRGenSQLResultsLayer in ogr_gensql.cpp and set the field
264type properly:
265
266::
267
268   else if ( psColDef->field_index >= iFIDFieldIndex )
269   {
270       switch ( SpecialFieldTypes[psColDef->field_index - iFIDFieldIndex] )
271       {
272       case SWQ_INTEGER:
273           oFDefn.SetType( OFTInteger );
274           break;
275       case SWQ_STRING:
276           oFDefn.SetType( OFTString );
277           break;
278       case SWQ_FLOAT:
279           oFDefn.SetType( OFTReal );
280           break;
281       }
282   }
283
284Some of the queries will require to modify
285OGRGenSQLResultsLayer::PrepareSummary in ogr_gensql.cpp will be
286simplified (GetFieldAsString will be used in all cases to access the
287field values):
288
289::
290
291   pszError = swq_select_summarize( psSelectInfo, iField,
292   poSrcFeature->GetFieldAsString( psColDef->field_index ) );
293
294OGRGenSQLResultsLayer::TranslateFeature should also be modifyed when
295copying the fields from primary record to the destination feature
296
297::
298
299    if ( psColDef->field_index >= iFIDFieldIndex &&
300               psColDef->field_index < iFIDFieldIndex + SPECIAL_FIELD_COUNT )
301   {
302       switch (SpecialFieldTypes[psColDef->field_index - iFIDFieldIndex])
303       {
304       case SWQ_INTEGER:
305           poDstFeat->SetField( iField, poSrcFeat->GetFieldAsInteger(psColDef->field_index) );
306       case SWQ_STRING:
307           poDstFeat->SetField( iField, poSrcFeat->GetFieldAsString(psColDef->field_index) );
308       }
309   }
310
311For supporting the 'order by' queries we should also modify
312OGRGenSQLResultsLayer::CreateOrderByIndex() as:
313
314::
315
316
317   if ( psKeyDef->field_index >= iFIDFieldIndex)
318   {
319       if ( psKeyDef->field_index < iFIDFieldIndex + SPECIAL_FIELD_COUNT )
320       {
321           switch (SpecialFieldTypes[psKeyDef->field_index - iFIDFieldIndex])
322           {
323           case SWQ_INTEGER:
324               psDstField->Integer = poSrcFeat->GetFieldAsInteger(psKeyDef->field_index);
325           case SWQ_STRING:
326               psDstField->String = CPLStrdup( poSrcFeat->GetFieldAsString(psKeyDef->field_index) );
327           }
328       }
329       continue;
330   }
331
332All of the strings allocated previously should be deallocated later in
333the same function as:
334
335::
336
337
338   if ( psKeyDef->field_index >= iFIDFieldIndex )
339   {
340       /* warning: only special fields of type string should be deallocated */
341       if (SpecialFieldTypes[psKeyDef->field_index - iFIDFieldIndex] == SWQ_STRING)
342       {
343           for( i = 0; i < nIndexSize; i++ )
344           {
345               OGRField *psField = pasIndexFields + iKey + i * nOrderItems;
346               CPLFree( psField->String );
347           }
348       }
349       continue;
350   }
351
352When ordering by the field values the OGRGenSQLResultsLayer::Compare
353should also be modifyed:
354
355::
356
357   if( psKeyDef->field_index >= iFIDFieldIndex )
358       poFDefn = NULL;
359   else
360       poFDefn = poSrcLayer->GetLayerDefn()->GetFieldDefn(
361           psKeyDef->field_index );
362
363   if( (pasFirstTuple[iKey].Set.nMarker1 == OGRUnsetMarker
364           && pasFirstTuple[iKey].Set.nMarker2 == OGRUnsetMarker)
365       || (pasSecondTuple[iKey].Set.nMarker1 == OGRUnsetMarker
366           && pasSecondTuple[iKey].Set.nMarker2 == OGRUnsetMarker) )
367       nResult = 0;
368   else if ( poFDefn == NULL )
369   {
370       switch (SpecialFieldTypes[psKeyDef->field_index - iFIDFieldIndex])
371       {
372       case SWQ_INTEGER:
373           if( pasFirstTuple[iKey].Integer < pasSecondTuple[iKey].Integer )
374               nResult = -1;
375           else if( pasFirstTuple[iKey].Integer > pasSecondTuple[iKey].Integer )
376               nResult = 1;
377           break;
378       case SWQ_STRING:
379           nResult = strcmp(pasFirstTuple[iKey].String,
380                           pasSecondTuple[iKey].String);
381           break;
382       }
383   }
384
385Adding New Special Fields
386-------------------------
387
388Adding a new special field in a subsequent development phase is fairly
389straightforward and the following steps should be made:
390
3911. In ogr_p.h a new constant should be added with the value of the
392   SPECIAL_FIELD_COUNT and SPECIAL_FIELD_COUNT should be incremented by
393   one.
394
3952. In ogrfeaturequery.cpp the special field string and the type should
396   be added to SpecialFieldNames and SpecialFieldTypes respectively
397
3983. The field value accessors (OGRFeature::GetFieldAsString,
399   OGRFeature::GetFieldAsInteger, OGRFeature::GetFieldAsDouble) should
400   be modifyed to provide the value of the new special field. All of
401   these functions provide const return values so GetFieldAsString
402   should retain the value in the m_pszTmpFieldValue member.
403
4044. When adding a new value with a type other than SWQ_INTEGER and
405   SWQ_STRING the following functions might also be modified
406   accordingly:
407
408-  OGRGenSQLResultsLayer::OGRGenSQLResultsLayer
409-  OGRGenSQLResultsLayer::TranslateFeature
410-  OGRGenSQLResultsLayer::CreateOrderByIndex
411-  OGRGenSQLResultsLayer::Compare
412-  OGRFeatureQueryEvaluator
413
414Backward Compatibility
415----------------------
416
417In most cases the backward compatibility of the OGR library will be
418retained. However the special fields will potentially conflict with
419regard fields with the given names. When accessing the field values the
420special fields will take pecedence over the other fields with the same
421names.
422
423When using OGRFeature::GetFieldAsString the returned value will be
424stored as a member variable instead of a static variable. The string
425will be deallocated and will no longer be usable after the destruction
426of the feature.
427
428Regression Testing
429------------------
430
431A new gdalautotest/ogr/ogr_sqlspecials.py script to test support for all
432special fields in the ExecuteSQL() call and with WHERE clauses.
433
434Documentation
435-------------
436
437The OGR SQL document will be updated to reflect the support for special
438fields.
439
440Implementation Staffing
441-----------------------
442
443Tamas Szekeres will implement the bulk of the RFC in time for GDAL/OGR
4441.4.0.
445
446Frank Warmerdam will consider how the backward compatibility issues
447(with special regard to the modified lifespan of the GetFieldAsString
448returned value) will affect the other parts of the OGR project and will
449write the Python regression testing script.
450
451References
452----------
453
454-  Tracking bug for this feature (containing all of the proposed code
455   changes): #1333
456
457-  MapServer related bugs:
458
459   -  `1129 <http://trac.osgeo.org/mapserver/ticket/1129>`__
460   -  `1438 <http://trac.osgeo.org/mapserver/ticket/1438>`__
461
462Voting History
463--------------
464
465Frank Warmerdam +1
466
467Daniel Morissette +1
468
469Howard Butler +0
470
471Andrey Kiselev +1
472