1.. _rfc-78: 2 3=========================================== 4RFC 78: gdal-utils package 5=========================================== 6 7============== ============================ 8Author: Idan Miara 9Contact: idan@miara.com 10Started: 2020-Dec-3 11Last updated: 2021-March-26 12Status: Adopted, implemented in GDAL 3.3 13============== ============================ 14 15Summary 16------- 17 18This RFC suggests to put all the GDAL python modules (formly scripts), except from the GDAL core SWIG bindings, 19into their own distribution on pypi. 20The GDAL python sub-package `osgeo.utils` (introduced in GDAL 3.2) would be renamed into a package named `osgeo_utils`. 21 22The standalone python scripts from GDAL <= 3.1 were transformed to `osgeo.utils` in GDAL 3.2. 23For backwards compatibility these scripts still exist and function as tiny wrappers around the python modules. 24Users of these scripts would not be effected from this RFC as the scripts would continue to function in GDAL 3.3 25in the same way as in GDAL <= 3.2. 26 27To allow maximum backwards compatibility, The `osgeo` package (which includes the GDAL core SWIG bindings) 28and the `osgeo_utils` package will continue to be distributed in a single `sdist` named `gdal` in `pypi`. 29 30In addition, a new pure python `wheel` distribution named `gdal-utils` will be available in `pypi` under the name 31`gdal-utils`. 32 33This will allow users who wish to upgrade the utils without upgrading the bindings to do so with 34`pip install --upgrade gdal-utils` (see more details in the following sections). 35 36Motivation 37---------- 38 39Making gdal Python developers life easier on Windows (and maybe other platforms): 40 The straightforward way of cloning `gdal` and adding `gdal/swig/python` to Python path 41 (In PyCharm: marking the it as a `Source Root`) won't work because the `pyc` files are missing from 42 `gdal/swig/python/osgeo`, thus by adding `osgeo` to Python path we would be masking a binary installation of gdal 43 that might be already installed (i.e. `osgeo4w` or Christoph Gohlke's binary Windows wheels). 44 Workarounds, like copying the `pyc` files to the `osgeo` dir, cause their own problems, like: 45 * Switching interpreters that have different versions of `gdal` causes more problems. 46 * Non clean git working tree so committing the changes to git is harder. 47 By moving the `gdal-utils` into another root this problem is completely avoid. 48 49Allow mixing `gdal` and `gdal-utils` versions 50 As the Python code evolves semi-independently of the GDAL core and is not directly dependent on a specific 51 GDAL version, one might want use the latest `gdal-utils` package with an older version of `gdal` core bindings, 52 or vice versa. Currently, One would need to mix the contents different `gdal` packages to do so. 53 54 As the `gdal` package is platform specific, and requires compilation, In some distributions upgrading 55 to a new GDAL version might take more time, so one could upgrade the `gdal-utils` package easily 56 and independently of upgrading the `gdal` package, using `pip install --upgrade gdal-utils`. 57 58 On the other hand, if we drop another Python version, like we just did with Python 2.7, 59 One might be still be able to use newer gdal core bindings (which might still support an older version of Python) 60 with an older `gdal-utils`, Which might drop Python versions sooner. 61 62 Reasons why users wouldn't be able to upgrade to recent GDAL: 63 64 * They use some LTS distribution (like Debian), or application (like QGIS LTS). 65 * Recent GDAL is not available for their platform or distribution (even if not LTS). 66 * Their code dependents on some binary gdal-plugin which is available for a specific GDAL version. 67 68 A concrete example would be someone who uses QGIS (or QGIS LTS) that currently comes with GDAL 3.1, 69 and wants to use the neat stuff added to `gdal_calc`, With this RFC he could just `pip install --upgrade gdal-utils`. 70 Because this package package is Pure Python, upgrades should be easy. 71 72 Although each version of `gdal-utils` would only be tested against the equivalent version of `gdal`, 73 In most cases different versions would still be compatible, but without guarantee. 74 75 This RFC would make further testing the compatibility between different versions of these packages easier and 76 could be considered the first step into making the `gdal-utils` package completely independent of the `gdal` package. 77 It would also potentially allow forward compatibility in case we ever decide to remove the utils from the `gdal` wheel 78 and keep them only in the `gdal-utils` wheel. 79 80 81Package Names and PyPi release: 82---------------------------------- 83 84This RFC suggests to keep the all-in-one package and in addition to distribute a utils only package, 85In order to keep the required changes from users to a minimum and at the same time allow mixing versions: 86 87* https://pypi.org/project/GDAL/ 88 89Distribution of a single `pypi.org` wheel named `gdal` will be retained and will includes both packages. 90This will insure smooth transition and maximum backwards compatibility. 91 92* https://pypi.org/project/gdal-utils/ 93 94A new `pypi.org` package is introduced to include with just the `osgeo_utils` package, which will allow 95upgrading the utils without upgrading the bindings. 96 97To be consistent with the `gdal` `pypi` package name, the utils `pypi` package is named `gdal-utils`. 98 99the name `osgeo_utils` is consistent with the `osgeo` namespace and module names. 100 101.. code-block:: 102 103 pip install gdal 104 pip install gdal-utils 105 106.. code-block:: Python 107 108 from osgeo import gdal 109 from osgeo_utils import gdal_calc, ogr_foo, osr_bar 110 111 112 113How to upgrade the utils without upgrading the bindings: 114---------------------------------------------------------- 115 116> If someone installs the "gdal" all-in-one package and the "gdal-utils" one. Wouldn't that conflict ? 117 118`pip install` a wheel overwrites whichever files already exist (even if installed by a different package) 119If you `pip install gdal` then `pip install gdal-utils` you'd get the utils from `gdal-utils`. 120If later you do again `pip install gdal` with a different version then you'd get the utils from `gdal` again, and so on. 121(it doesn't seem that it matters which version is a bigger number, just which one you installed later) 122 123If you `pip install gdal` then `pip install gdal-utils` and then `pip uninstall gdal-utils` then the 124utils would be uninstalled and you'd be left with gdal without utils. 125Then you could `pip install gdal-utils` or `pip install gdal --ignore-installed` to get them back again 126(`--ignore-installed` is not required if you install a different version) 127 128Limitations and scope 129--------------------- 130 131The scope of this RFC is the GDAL Python code, except for the SWIG bindings. 132There is no effect on any other language supported by GDAL. 133Because gdal core is tested using the Python SWIG bindings - this RFC does not suggest changing them in any way. 134Binary wheel distribution - discussed in the past and related to ideas in this RFC. 135 136 137`gdal` and `gdal-utils` Compatibility 138---------------------------------------- 139 140This RFC suggests that `gdal-utils` would continue to be only tested against the same version of `gdal`. 141In most cases different versions would still be compatible, but without guarantee. 142 143A minimum energy approach might keep `gdal-utils` compatible with some `gdal` versions != `x.y`. 144`gdal-utils` might officially drop support of some too old version of GDAL 145by specifying a newer minimum version of GDAL in the `setup.py` of `gdal-utils`. 146 147For maximum backwards compatibility and because we would only test `gdal` against the same version of `gdal-utils` - 148`osgeo` and `osgeo_utils` will continue to be distributed inside a single wheel in addition 149to the new separate wheel for the utils only. 150 151In cases were an `gdal-utils` module or function actually does need a minimum specific version of `gdal` 152(i.e. dependence on a new GDAL C API) Compatibility could be checked at runtime by comparing to `osgeo.__version__`. 153 154Versioning 155------------ 156 157As development of `gdal-utils` will be still tied with the development of GDAL and will be 158released together with the same `x.y.z` version number. 159In case a hotfix to `gdal-utils` is required for some reason, a `x.y.z.p` version might be used, 160Which will not effect the distribution of version `z+1`, i.e. `3.3.0` < `3.3.0.1` < `3.3.1`. 161 162 163Backward compatibility issues: 164-------------------------------- 165 166* `osgeo.utils` will need be replaced with `osgeo_utils` 167 This is the only breaking change, only for GDAL=3.2, and only a single character. 168* `swig/python/scripts` - users of the gdal scripts (which are thin wrappers around the utils) wouldn't be effected. 169 170 171Folder structure change 172-------------------------- 173 174* `gdal/swig/python/osgeo/utils` -> `gdal/swig/python/gdal-utils/osgeo_utils` 175 176* `gdal/swig/python/osgeo/setup.py` - was updated to include the utils from the new location under the `gdal-utils` folder. 177 178* `gdal/swig/python/gdal-utils/setup.py` - additional setup was added for `gdal-utils`. 179 180CI Impacts: 181------------ 182 183`gdal-utils` wheel building could be added to the CI, i.e. like in https://github.com/OSGeo/gdal/pull/3579 184No other CI Impacts. 185 186Impacts on GDAL core 187-------------------- 188 189None. 190 191SWIG binding changes 192-------------------- 193 194None. 195 196Security implications 197--------------------- 198 199None. 200 201Performance impact 202------------------ 203 204None. 205 206Documentation 207------------- 208 209Implications of this change shell documented in the README. 210 211Testing 212------- 213 214Minor changes were made to pytest. 215 216Previous discussions 217-------------------- 218 219This topic has been discussed in the past in : 220 221- http://osgeo-org.1560.x6.nabble.com/gdal-dev-Call-for-discussion-on-RFC77-Drop-Python-2-support-td5449659.html 222 223Related PRs: 224------------- 225 226- https://gdal.org/development/rfc/rfc77_drop_python2_support.html 227- https://github.com/OSGeo/gdal/pull/3131 228- https://github.com/OSGeo/gdal/pull/3117 229- https://github.com/OSGeo/gdal/pull/3247 230 231Voting history 232-------------- 233 234- http://osgeo-org.1560.x6.nabble.com/gdal-dev-Motion-RFC-78-gdal-utils-package-td5482707.html 235 236* +1 from EvenR, HowardB 237* +0 from KurtS, JukkaR 238* -0 from SeanG 239 240Credits 241------- 242 243* Implemented by the author of this RFC, Idan Miara. 244