1.. _rfc-60:
2
3=======================================================================================
4RFC 60 : Improved round-tripping in OGR
5=======================================================================================
6
7Author: Even Rouault
8
9Contact: even.rouault at spatialys.com
10
11Status: Adopted, implemented
12
13Implementation version: 2.1
14
15Summary
16-------
17
18This RFC defines how to improve better round-tripping in conversion of
19vector formats, in particular for GeoJSON extensions.
20
21Rationale
22---------
23
24Some formats have concepts that are not well modeled by the OGR
25abstraction, but that are desirable to be preserved in transformation
26scenarios involving reprojection, spatial/attribute filtering, clipping,
27etc... where the target format is the source format.
28
29Various extensions exist above the core GeoJSON specification: at the
30FeatureCollection, Feature or Geometry levels.
31
32See
33`https://github.com/mapbox/carmen/blob/master/carmen-geojson.md <https://github.com/mapbox/carmen/blob/master/carmen-geojson.md>`__,
34
35::
36
37   {
38       "type": "FeatureCollection",
39       "query": ["austin"],
40       "features": [
41           {
42               "type": "Feature",
43               "id": "place.4201",
44               "text": "Austin",
45               "place_name": "Austin, Texas, United States",
46               "bbox": [-97.9383829999999, 30.098659, -97.5614889999999, 30.516863],
47               "center": [-97.7559964, 30.3071816],
48               "geometry": {
49                   "type": "Point",
50                   "coordinates": [-97.7559964, 30.3071816]
51               },
52               "properties": {
53                   "title": "Austin",
54                   "type": "city",
55                   "score": 600000790107194.8
56               },
57               "context": [
58                   {
59                       "id": "province.293",
60                       "text": "Texas"
61                   },
62                   {
63                       "id": "country.51",
64                       "text": "United States"
65                   }
66               ]
67           },
68           ...
69       ]
70   }
71
72`https://github.com/geocoders/geocodejson-spec/blob/master/draft/README.md <https://github.com/geocoders/geocodejson-spec/blob/master/draft/README.md>`__:
73
74::
75
76   {
77
78     // REQUIRED. GeocodeJSON result is a FeatureCollection.
79     "type": "FeatureCollection",
80
81     // REQUIRED. Namespace.
82     "geocoding": {
83
84       // REQUIRED. A semver.org compliant version number. Describes the version of
85       // the GeocodeJSON spec that is implemented by this instance.
86       "version": "0.1.0",
87
88       // OPTIONAL. Default: null. The licence of the data. In case of multiple sources,
89       // and then multiple licences, can be an object with one key by source.
90       "licence": "ODbL",
91
92       // OPTIONAL. Default: null. The attribution of the data. In case of multiple sources,
93       // and then multiple attributions, can be an object with one key by source.
94       "attribution": "OpenStreetMap Contributors",
95
96       // OPTIONAL. Default: null. The query that has been issued to trigger the
97       // search.
98       "query": "24 allée de Bercy 75012 Paris",
99
100     },
101
102     // REQUIRED. As per GeoJSON spec.
103     "features": [
104       // OPTIONAL. An array of feature objects. See below.
105     ]
106   }
107
108or
109`https://github.com/geojson/draft-geojson/issues/80#issuecomment-138037554 <https://github.com/geojson/draft-geojson/issues/80#issuecomment-138037554>`__
110for a few examples.
111
112::
113
114   { "type" : "GeometryCollection",
115     "geometries" : [
116       { "type" : "LineString",
117         "extensions" : [ "time", "atemp", "hr", "cad" ],
118         "coordinates" : [
119           [
120             -122.45671039447188,
121             37.786870915442705,
122             0.4000000059604645,
123             "2014-11-06T19:16:06.000Z",
124             31.0,
125             99,
126             0
127           ],
128
129Changes
130-------
131
132OGRFeature
133~~~~~~~~~~
134
135Two new members will be added to the OGRFeature class, m_pszNativeData
136(string) and m_pszNativeMediaType (string). m_pszNativeData will contain
137the representation (or part of the representation) of the original
138feature, and m_pszNativeMediaType the `media
139type <https://en.wikipedia.org/wiki/Media_type>`__
140
141The following methods will be added to OGRFeature class:
142
143::
144
145   public:
146       const char *GetNativeData() const;
147       const char *GetNativeMediaType() const;
148       void        SetNativeData( const char* pszNativeData );
149       void        SetNativeMediaType( const char* pszNativeMediaType );
150
151Thus, in the GeoJSON case, nativeData would contain the full
152serialization of a GeoJSON Feature. m_pszNativeMediaType would be set to
153"application/vnd.geo+json" The writer side of the GeoJSON driver would
154start from the nativeData if present (and if nativeMediaType =
155"application/vnd.geo+json", replace its properties member with the
156content of the OGR fields and patch its geometry to include additional
157JSON objects.
158
159The OGRFeature::Clone() and ::SetFrom() methods will propagate
160nativeData and nativeMediaType.
161
162OGRLayer
163~~~~~~~~
164
165A dedicated metadata domain "NATIVE_DATA" in which there would be a
166"NATIVE_DATA" and "NATIVE_MEDIA_TYPE" items would be used. In the
167GeoJSON case, this would contain JSON members at the FeatureCollection
168level (excluding the features array of course).
169
170Driver open options and layer creation options
171~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
172
173Drivers that support nativeData on read should expose a NATIVE_DATA
174boolean open option, and disable it by default so as not to impact
175performance. ogr2ogr will by default turn this option on.
176
177Drivers that support nativeData on write at the layer level should
178expose a NATIVE_DATA string and NATIVE_MEDIA_TYPE string layer creation
179options, so that ogr2ogr can fill them with the content of the
180NATIVE_DATA metadata domain of the source layer(s).
181
182C API
183-----
184
185The following functions will be added:
186
187::
188
189   const char CPL_DLL *OGR_F_GetNativeData(OGRFeatureH);
190   void OGR_F_SetNativeData(OGRFeatureH, const char*);
191   const char CPL_DLL *OGR_F_GetNativeMediaType(OGRFeatureH);
192   void OGR_F_SetNativeMediaType(OGRFeatureH, const char*);
193
194SQL result layers
195-----------------
196
197Both OGR SQL and SQLite SQL dialect implementations have been modified
198to propagate the content of the NATIVE_DATA metadata domain of the
199source layer (the one of the FROM table) to the target layer, and
200NativeData and NativeMediaType from source features are copied into
201target features.
202
203SWIG bindings (Python / Java / C# / Perl) changes
204-------------------------------------------------
205
206The new functions will mapped to SWIG as GetNativeData(),
207SetNativeData(), GetNativeMediaType() and SetNativeMediaType()
208
209Drivers
210-------
211
212The GeoJSON driver will be modified to implement this RFC, in read and
213write, and thus will\* :
214
215-  declare a NATIVE_DATA open option to enable storing layer and feature
216   native data.
217-  and NATIVE_DATA & NATIVE_MEDIA_TYPE layer creation options so as to
218   be able to write native data at FeatureCollection levels
219-  use OGRFeature nativeData on write.
220
221The effect of this is that ogr2ogr will be able to preserve the members
222marked between ``***`` in the below snippet:
223
224::
225
226   {
227     "type": "FeatureCollection",
228     ***"extra_fc_member": "foo",***
229     "features":
230     [
231       {
232         "type": "Feature",
233         ***"extra_feat_member": "bar",***
234         "geometry": {
235           "type": "Point",
236           ***extra_geom_member": "baz",***
237           "coordinates": [ 2, 49, 3, ***100, 101*** ]
238         },
239         "properties": {
240           "a_property": "foo",
241         }
242       }
243     ]
244   }
245
246Other drivers like ElasticSearch and MongoDB drivers, that use a \_json
247OGR field for round-tripping could potentially be upgraded to benefit
248from the mechanism of this RFC.
249
250Utilities
251---------
252
253ogr2ogr will be modified to automatically copy nativeData at layer and
254feature level. A -noNativeData flag will be added to avoid doing so,
255when this is not desirable.
256
257By default, ogr2ogr will open datasources with the NATIVE_DATA=YES open
258option so that drivers that can store nativeData do so. And if the
259output datasource supports the NATIVE_DATA and NATIVE_MEDIA_TYPE layer
260creation options, it will feel them with the content of the source layer
261NATIVE_DATA metadata domain.
262
263Documentation
264-------------
265
266All new methods/functions are documented.
267
268Test Suite
269----------
270
271The GeoJSON and ogr2ogr related tests will be extended
272
273Compatibility Issues
274--------------------
275
276Nothing severe expected. Potentially existing scripts might need to add
277-noNativeData to get previous behavior.
278
279Related ticket
280--------------
281
282`https://trac.osgeo.org/gdal/ticket/5310 <https://trac.osgeo.org/gdal/ticket/5310>`__
283
284Implementation
285--------------
286
287The implementation will be done by Even Rouault (Spatialys) and be
288sponsored by Mapbox.
289
290The proposed implementation lies in the "rfc60_native_data" branch of
291the
292​\ `https://github.com/rouault/gdal2/tree/rfc60_native_data <https://github.com/rouault/gdal2/tree/rfc60_native_data>`__,
293in pull request
294`https://github.com/OSGeo/gdal/pull/75 <https://github.com/OSGeo/gdal/pull/75>`__
295
296Voting history
297--------------
298
299+1 from HowardB, KurtS, TamasS, JukkaR and EvenR
300