1.. _rfc-78:
2
3===========================================
4RFC 78: gdal-utils package
5===========================================
6
7============== ============================
8Author:        Idan Miara
9Contact:       idan@miara.com
10Started:       2020-Dec-3
11Last updated:  2021-March-26
12Status:        Adopted, implemented in GDAL 3.3
13============== ============================
14
15Summary
16-------
17
18This RFC suggests to put all the GDAL python modules (formly scripts), except from the GDAL core SWIG bindings,
19into their own distribution on pypi.
20The GDAL python sub-package `osgeo.utils` (introduced in GDAL 3.2) would be renamed into a package named `osgeo_utils`.
21
22The standalone python scripts from GDAL <= 3.1 were transformed to `osgeo.utils` in GDAL 3.2.
23For backwards compatibility these scripts still exist and function as tiny wrappers around the python modules.
24Users of these scripts would not be effected from this RFC as the scripts would continue to function in GDAL 3.3
25in the same way as in GDAL <= 3.2.
26
27To allow maximum backwards compatibility, The `osgeo` package (which includes the GDAL core SWIG bindings)
28and the `osgeo_utils` package will continue to be distributed in a single `sdist` named `gdal` in `pypi`.
29
30In addition, a new pure python `wheel` distribution named `gdal-utils` will be available in `pypi` under the name
31`gdal-utils`.
32
33This will allow users who wish to upgrade the utils without upgrading the bindings to do so with
34`pip install --upgrade gdal-utils` (see more details in the following sections).
35
36Motivation
37----------
38
39Making gdal Python developers life easier on Windows (and maybe other platforms):
40    The straightforward way of cloning `gdal` and adding `gdal/swig/python` to Python path
41    (In PyCharm: marking the it as a `Source Root`) won't work because the `pyc` files are missing from
42    `gdal/swig/python/osgeo`, thus by adding `osgeo` to Python path we would be masking a binary installation of gdal
43    that might be already installed (i.e. `osgeo4w` or Christoph Gohlke's binary Windows wheels).
44    Workarounds, like copying the `pyc` files to the `osgeo` dir, cause their own problems, like:
45    * Switching interpreters that have different versions of `gdal` causes more problems.
46    * Non clean git working tree so committing the changes to git is harder.
47    By moving the `gdal-utils` into another root this problem is completely avoid.
48
49Allow mixing `gdal` and `gdal-utils` versions
50    As the Python code evolves semi-independently of the GDAL core and is not directly dependent on a specific
51    GDAL version, one might want use the latest `gdal-utils` package with an older version of `gdal` core bindings,
52    or vice versa. Currently, One would need to mix the contents different `gdal` packages to do so.
53
54    As the `gdal` package is platform specific, and requires compilation, In some distributions upgrading
55    to a new GDAL version might take more time, so one could upgrade the `gdal-utils` package easily
56    and independently of upgrading the `gdal` package, using `pip install --upgrade gdal-utils`.
57
58    On the other hand, if we drop another Python version, like we just did with Python 2.7,
59    One might be still be able to use newer gdal core bindings (which might still support an older version of Python)
60    with an older `gdal-utils`, Which might drop Python versions sooner.
61
62    Reasons why users wouldn't be able to upgrade to recent GDAL:
63
64    * They use some LTS distribution (like Debian), or application (like QGIS LTS).
65    * Recent GDAL is not available for their platform or distribution (even if not LTS).
66    * Their code dependents on some binary gdal-plugin which is available for a specific GDAL version.
67
68    A concrete example would be someone who uses QGIS (or QGIS LTS) that currently comes with GDAL 3.1,
69    and wants to use the neat stuff added to `gdal_calc`, With this RFC he could just `pip install --upgrade gdal-utils`.
70    Because this package package is Pure Python, upgrades should be easy.
71
72    Although each version of `gdal-utils` would only be tested against the equivalent version of `gdal`,
73    In most cases different versions would still be compatible, but without guarantee.
74
75    This RFC would make further testing the compatibility between different versions of these packages easier and
76    could be considered the first step into making the `gdal-utils` package completely independent of the `gdal` package.
77    It would also potentially allow forward compatibility in case we ever decide to remove the utils from the `gdal` wheel
78    and keep them only in the `gdal-utils` wheel.
79
80
81Package Names and PyPi release:
82----------------------------------
83
84This RFC suggests to keep the all-in-one package and in addition to distribute a utils only package,
85In order to keep the required changes from users to a minimum and at the same time allow mixing versions:
86
87* https://pypi.org/project/GDAL/
88
89Distribution of a single `pypi.org` wheel named `gdal` will be retained and will includes both packages.
90This will insure smooth transition and maximum backwards compatibility.
91
92* https://pypi.org/project/gdal-utils/
93
94A new `pypi.org` package is introduced to include with just the `osgeo_utils` package, which will allow
95upgrading the utils without upgrading the bindings.
96
97To be consistent with the `gdal` `pypi` package name, the utils `pypi` package is named `gdal-utils`.
98
99the name `osgeo_utils` is consistent with the `osgeo` namespace and module names.
100
101.. code-block::
102
103    pip install gdal
104    pip install gdal-utils
105
106.. code-block:: Python
107
108    from osgeo import gdal
109    from osgeo_utils import gdal_calc, ogr_foo, osr_bar
110
111
112
113How to upgrade the utils without upgrading the bindings:
114----------------------------------------------------------
115
116> If someone installs the "gdal" all-in-one package and the "gdal-utils" one. Wouldn't that conflict ?
117
118`pip install` a wheel overwrites whichever files already exist (even if installed by a different package)
119If you `pip install gdal` then `pip install gdal-utils` you'd get the utils from `gdal-utils`.
120If later you do again `pip install gdal` with a different version then you'd get the utils from `gdal` again, and so on.
121(it doesn't seem that it matters which version is a bigger number, just which one you installed later)
122
123If you `pip install gdal` then `pip install gdal-utils` and then `pip uninstall gdal-utils` then the
124utils would be uninstalled and you'd be left with gdal without utils.
125Then you could `pip install gdal-utils` or `pip install gdal --ignore-installed` to get them back again
126(`--ignore-installed` is not required if you install a different version)
127
128Limitations and scope
129---------------------
130
131The scope of this RFC is the GDAL Python code, except for the SWIG bindings.
132There is no effect on any other language supported by GDAL.
133Because gdal core is tested using the Python SWIG bindings - this RFC does not suggest changing them in any way.
134Binary wheel distribution - discussed in the past and related to ideas in this RFC.
135
136
137`gdal` and `gdal-utils` Compatibility
138----------------------------------------
139
140This RFC suggests that `gdal-utils` would continue to be only tested against the same version of `gdal`.
141In most cases different versions would still be compatible, but without guarantee.
142
143A minimum energy approach might keep `gdal-utils` compatible with some `gdal` versions != `x.y`.
144`gdal-utils` might officially drop support of some too old version of GDAL
145by specifying a newer minimum version of GDAL in the `setup.py` of `gdal-utils`.
146
147For maximum backwards compatibility and because we would only test `gdal` against the same version of `gdal-utils` -
148`osgeo` and `osgeo_utils` will continue to be distributed inside a single wheel in addition
149to the new separate wheel for the utils only.
150
151In cases were an `gdal-utils` module or function actually does need a minimum specific version of `gdal`
152(i.e. dependence on a new GDAL C API) Compatibility could be checked at runtime by comparing to `osgeo.__version__`.
153
154Versioning
155------------
156
157As development of `gdal-utils` will be still tied with the development of GDAL and will be
158released together with the same `x.y.z` version number.
159In case a hotfix to `gdal-utils` is required for some reason, a `x.y.z.p` version might be used,
160Which will not effect the distribution of version `z+1`, i.e. `3.3.0` < `3.3.0.1` < `3.3.1`.
161
162
163Backward compatibility issues:
164--------------------------------
165
166* `osgeo.utils` will need be replaced with `osgeo_utils`
167    This is the only breaking change, only for GDAL=3.2, and only a single character.
168* `swig/python/scripts` - users of the gdal scripts (which are thin wrappers around the utils) wouldn't be effected.
169
170
171Folder structure change
172--------------------------
173
174* `gdal/swig/python/osgeo/utils` -> `gdal/swig/python/gdal-utils/osgeo_utils`
175
176* `gdal/swig/python/osgeo/setup.py` - was updated to include the utils from the new location under the `gdal-utils` folder.
177
178* `gdal/swig/python/gdal-utils/setup.py` - additional setup was added for `gdal-utils`.
179
180CI Impacts:
181------------
182
183`gdal-utils` wheel building could be added to the CI, i.e. like in https://github.com/OSGeo/gdal/pull/3579
184No other CI Impacts.
185
186Impacts on GDAL core
187--------------------
188
189None.
190
191SWIG binding changes
192--------------------
193
194None.
195
196Security implications
197---------------------
198
199None.
200
201Performance impact
202------------------
203
204None.
205
206Documentation
207-------------
208
209Implications of this change shell documented in the README.
210
211Testing
212-------
213
214Minor changes were made to pytest.
215
216Previous discussions
217--------------------
218
219This topic has been discussed in the past in :
220
221- http://osgeo-org.1560.x6.nabble.com/gdal-dev-Call-for-discussion-on-RFC77-Drop-Python-2-support-td5449659.html
222
223Related PRs:
224-------------
225
226- https://gdal.org/development/rfc/rfc77_drop_python2_support.html
227- https://github.com/OSGeo/gdal/pull/3131
228- https://github.com/OSGeo/gdal/pull/3117
229- https://github.com/OSGeo/gdal/pull/3247
230
231Voting history
232--------------
233
234- http://osgeo-org.1560.x6.nabble.com/gdal-dev-Motion-RFC-78-gdal-utils-package-td5482707.html
235
236* +1 from EvenR, HowardB
237* +0 from KurtS, JukkaR
238* -0 from SeanG
239
240Credits
241-------
242
243* Implemented by the author of this RFC, Idan Miara.
244