1.. _rfc-60: 2 3======================================================================================= 4RFC 60 : Improved round-tripping in OGR 5======================================================================================= 6 7Author: Even Rouault 8 9Contact: even.rouault at spatialys.com 10 11Status: Adopted, implemented 12 13Implementation version: 2.1 14 15Summary 16------- 17 18This RFC defines how to improve better round-tripping in conversion of 19vector formats, in particular for GeoJSON extensions. 20 21Rationale 22--------- 23 24Some formats have concepts that are not well modeled by the OGR 25abstraction, but that are desirable to be preserved in transformation 26scenarios involving reprojection, spatial/attribute filtering, clipping, 27etc... where the target format is the source format. 28 29Various extensions exist above the core GeoJSON specification: at the 30FeatureCollection, Feature or Geometry levels. 31 32See 33`https://github.com/mapbox/carmen/blob/master/carmen-geojson.md <https://github.com/mapbox/carmen/blob/master/carmen-geojson.md>`__, 34 35:: 36 37 { 38 "type": "FeatureCollection", 39 "query": ["austin"], 40 "features": [ 41 { 42 "type": "Feature", 43 "id": "place.4201", 44 "text": "Austin", 45 "place_name": "Austin, Texas, United States", 46 "bbox": [-97.9383829999999, 30.098659, -97.5614889999999, 30.516863], 47 "center": [-97.7559964, 30.3071816], 48 "geometry": { 49 "type": "Point", 50 "coordinates": [-97.7559964, 30.3071816] 51 }, 52 "properties": { 53 "title": "Austin", 54 "type": "city", 55 "score": 600000790107194.8 56 }, 57 "context": [ 58 { 59 "id": "province.293", 60 "text": "Texas" 61 }, 62 { 63 "id": "country.51", 64 "text": "United States" 65 } 66 ] 67 }, 68 ... 69 ] 70 } 71 72`https://github.com/geocoders/geocodejson-spec/blob/master/draft/README.md <https://github.com/geocoders/geocodejson-spec/blob/master/draft/README.md>`__: 73 74:: 75 76 { 77 78 // REQUIRED. GeocodeJSON result is a FeatureCollection. 79 "type": "FeatureCollection", 80 81 // REQUIRED. Namespace. 82 "geocoding": { 83 84 // REQUIRED. A semver.org compliant version number. Describes the version of 85 // the GeocodeJSON spec that is implemented by this instance. 86 "version": "0.1.0", 87 88 // OPTIONAL. Default: null. The licence of the data. In case of multiple sources, 89 // and then multiple licences, can be an object with one key by source. 90 "licence": "ODbL", 91 92 // OPTIONAL. Default: null. The attribution of the data. In case of multiple sources, 93 // and then multiple attributions, can be an object with one key by source. 94 "attribution": "OpenStreetMap Contributors", 95 96 // OPTIONAL. Default: null. The query that has been issued to trigger the 97 // search. 98 "query": "24 allée de Bercy 75012 Paris", 99 100 }, 101 102 // REQUIRED. As per GeoJSON spec. 103 "features": [ 104 // OPTIONAL. An array of feature objects. See below. 105 ] 106 } 107 108or 109`https://github.com/geojson/draft-geojson/issues/80#issuecomment-138037554 <https://github.com/geojson/draft-geojson/issues/80#issuecomment-138037554>`__ 110for a few examples. 111 112:: 113 114 { "type" : "GeometryCollection", 115 "geometries" : [ 116 { "type" : "LineString", 117 "extensions" : [ "time", "atemp", "hr", "cad" ], 118 "coordinates" : [ 119 [ 120 -122.45671039447188, 121 37.786870915442705, 122 0.4000000059604645, 123 "2014-11-06T19:16:06.000Z", 124 31.0, 125 99, 126 0 127 ], 128 129Changes 130------- 131 132OGRFeature 133~~~~~~~~~~ 134 135Two new members will be added to the OGRFeature class, m_pszNativeData 136(string) and m_pszNativeMediaType (string). m_pszNativeData will contain 137the representation (or part of the representation) of the original 138feature, and m_pszNativeMediaType the `media 139type <https://en.wikipedia.org/wiki/Media_type>`__ 140 141The following methods will be added to OGRFeature class: 142 143:: 144 145 public: 146 const char *GetNativeData() const; 147 const char *GetNativeMediaType() const; 148 void SetNativeData( const char* pszNativeData ); 149 void SetNativeMediaType( const char* pszNativeMediaType ); 150 151Thus, in the GeoJSON case, nativeData would contain the full 152serialization of a GeoJSON Feature. m_pszNativeMediaType would be set to 153"application/vnd.geo+json" The writer side of the GeoJSON driver would 154start from the nativeData if present (and if nativeMediaType = 155"application/vnd.geo+json", replace its properties member with the 156content of the OGR fields and patch its geometry to include additional 157JSON objects. 158 159The OGRFeature::Clone() and ::SetFrom() methods will propagate 160nativeData and nativeMediaType. 161 162OGRLayer 163~~~~~~~~ 164 165A dedicated metadata domain "NATIVE_DATA" in which there would be a 166"NATIVE_DATA" and "NATIVE_MEDIA_TYPE" items would be used. In the 167GeoJSON case, this would contain JSON members at the FeatureCollection 168level (excluding the features array of course). 169 170Driver open options and layer creation options 171~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 172 173Drivers that support nativeData on read should expose a NATIVE_DATA 174boolean open option, and disable it by default so as not to impact 175performance. ogr2ogr will by default turn this option on. 176 177Drivers that support nativeData on write at the layer level should 178expose a NATIVE_DATA string and NATIVE_MEDIA_TYPE string layer creation 179options, so that ogr2ogr can fill them with the content of the 180NATIVE_DATA metadata domain of the source layer(s). 181 182C API 183----- 184 185The following functions will be added: 186 187:: 188 189 const char CPL_DLL *OGR_F_GetNativeData(OGRFeatureH); 190 void OGR_F_SetNativeData(OGRFeatureH, const char*); 191 const char CPL_DLL *OGR_F_GetNativeMediaType(OGRFeatureH); 192 void OGR_F_SetNativeMediaType(OGRFeatureH, const char*); 193 194SQL result layers 195----------------- 196 197Both OGR SQL and SQLite SQL dialect implementations have been modified 198to propagate the content of the NATIVE_DATA metadata domain of the 199source layer (the one of the FROM table) to the target layer, and 200NativeData and NativeMediaType from source features are copied into 201target features. 202 203SWIG bindings (Python / Java / C# / Perl) changes 204------------------------------------------------- 205 206The new functions will mapped to SWIG as GetNativeData(), 207SetNativeData(), GetNativeMediaType() and SetNativeMediaType() 208 209Drivers 210------- 211 212The GeoJSON driver will be modified to implement this RFC, in read and 213write, and thus will\* : 214 215- declare a NATIVE_DATA open option to enable storing layer and feature 216 native data. 217- and NATIVE_DATA & NATIVE_MEDIA_TYPE layer creation options so as to 218 be able to write native data at FeatureCollection levels 219- use OGRFeature nativeData on write. 220 221The effect of this is that ogr2ogr will be able to preserve the members 222marked between ``***`` in the below snippet: 223 224:: 225 226 { 227 "type": "FeatureCollection", 228 ***"extra_fc_member": "foo",*** 229 "features": 230 [ 231 { 232 "type": "Feature", 233 ***"extra_feat_member": "bar",*** 234 "geometry": { 235 "type": "Point", 236 ***extra_geom_member": "baz",*** 237 "coordinates": [ 2, 49, 3, ***100, 101*** ] 238 }, 239 "properties": { 240 "a_property": "foo", 241 } 242 } 243 ] 244 } 245 246Other drivers like ElasticSearch and MongoDB drivers, that use a \_json 247OGR field for round-tripping could potentially be upgraded to benefit 248from the mechanism of this RFC. 249 250Utilities 251--------- 252 253ogr2ogr will be modified to automatically copy nativeData at layer and 254feature level. A -noNativeData flag will be added to avoid doing so, 255when this is not desirable. 256 257By default, ogr2ogr will open datasources with the NATIVE_DATA=YES open 258option so that drivers that can store nativeData do so. And if the 259output datasource supports the NATIVE_DATA and NATIVE_MEDIA_TYPE layer 260creation options, it will feel them with the content of the source layer 261NATIVE_DATA metadata domain. 262 263Documentation 264------------- 265 266All new methods/functions are documented. 267 268Test Suite 269---------- 270 271The GeoJSON and ogr2ogr related tests will be extended 272 273Compatibility Issues 274-------------------- 275 276Nothing severe expected. Potentially existing scripts might need to add 277-noNativeData to get previous behavior. 278 279Related ticket 280-------------- 281 282`https://trac.osgeo.org/gdal/ticket/5310 <https://trac.osgeo.org/gdal/ticket/5310>`__ 283 284Implementation 285-------------- 286 287The implementation will be done by Even Rouault (Spatialys) and be 288sponsored by Mapbox. 289 290The proposed implementation lies in the "rfc60_native_data" branch of 291the 292\ `https://github.com/rouault/gdal2/tree/rfc60_native_data <https://github.com/rouault/gdal2/tree/rfc60_native_data>`__, 293in pull request 294`https://github.com/OSGeo/gdal/pull/75 <https://github.com/OSGeo/gdal/pull/75>`__ 295 296Voting history 297-------------- 298 299+1 from HowardB, KurtS, TamasS, JukkaR and EvenR 300