1.. include:: global.rst.inc 2.. highlight:: bash 3.. _development: 4 5Development 6=========== 7 8This chapter will get you started with |project_name| development. 9 10|project_name| is written in Python (with a little bit of Cython and C for 11the performance critical parts). 12 13Contributions 14------------- 15 16... are welcome! 17 18Some guidance for contributors: 19 20- Discuss changes on the GitHub issue tracker, on IRC or on the mailing list. 21 22- Make your PRs on the ``master`` branch (see `Branching Model`_ for details). 23 24- Do clean changesets: 25 26 - Focus on some topic, resist changing anything else. 27 - Do not do style changes mixed with functional changes. 28 - Try to avoid refactorings mixed with functional changes. 29 - If you need to fix something after commit/push: 30 31 - If there are ongoing reviews: do a fixup commit you can 32 squash into the bad commit later. 33 - If there are no ongoing reviews or you did not push the 34 bad commit yet: amend the commit to include your fix or 35 merge the fixup commit before pushing. 36 - Have a nice, clear, typo-free commit comment. 37 - If you fixed an issue, refer to it in your commit comment. 38 - Follow the style guide (see below). 39 40- If you write new code, please add tests and docs for it. 41 42- Run the tests, fix any issues that come up. 43 44- Make a pull request on GitHub. 45 46- Wait for review by other developers. 47 48Branching model 49--------------- 50 51Borg development happens on the ``master`` branch and uses GitHub pull 52requests (if you don't have GitHub or don't want to use it you can 53send smaller patches via the borgbackup mailing list to the maintainers). 54 55Stable releases are maintained on maintenance branches named ``x.y-maint``, eg. 56the maintenance branch of the 1.0.x series is ``1.0-maint``. 57 58Most PRs should be filed against the ``master`` branch. Only if an 59issue affects **only** a particular maintenance branch a PR should be 60filed against it directly. 61 62While discussing / reviewing a PR it will be decided whether the 63change should be applied to maintenance branches. Each maintenance 64branch has a corresponding *backport/x.y-maint* label, which will then 65be applied. 66 67Changes that are typically considered for backporting: 68 69- Data loss, corruption and inaccessibility fixes. 70- Security fixes. 71- Forward-compatibility improvements. 72- Documentation corrections. 73 74.. rubric:: Maintainer part 75 76From time to time a maintainer will backport the changes for a 77maintenance branch, typically before a release or if enough changes 78were collected: 79 801. Notify others that you're doing this to avoid duplicate work. 812. Branch a backporting branch off the maintenance branch. 823. Cherry pick and backport the changes from each labelled PR, remove 83 the label for each PR you've backported. 84 85 To preserve authorship metadata, do not follow the ``git cherry-pick`` 86 instructions to use ``git commit`` after resolving conflicts. Instead, 87 stage conflict resolutions and run ``git cherry-pick --continue``, 88 much like using ``git rebase``. 89 90 To avoid merge issues (a cherry pick is a form of merge), use 91 these options (similar to the ``git merge`` options used previously, 92 the ``-x`` option adds a reference to the original commit):: 93 94 git cherry-pick --strategy recursive -X rename-threshold=5% -x 95 964. Make a PR of the backporting branch against the maintenance branch 97 for backport review. Mention the backported PRs in this PR, e.g.: 98 99 Includes changes from #2055 #2057 #2381 100 101 This way GitHub will automatically show in these PRs where they 102 were backported. 103 104.. rubric:: Historic model 105 106Previously (until release 1.0.10) Borg used a `"merge upwards" 107<https://git-scm.com/docs/gitworkflows#_merging_upwards>`_ model where 108most minor changes and fixes where committed to a maintenance branch 109(eg. 1.0-maint), and the maintenance branch(es) were regularly merged 110back into the main development branch. This became more and more 111troublesome due to merges growing more conflict-heavy and error-prone. 112 113Code and issues 114--------------- 115 116Code is stored on GitHub, in the `Borgbackup organization 117<https://github.com/borgbackup/borg/>`_. `Issues 118<https://github.com/borgbackup/borg/issues>`_ and `pull requests 119<https://github.com/borgbackup/borg/pulls>`_ should be sent there as 120well. See also the :ref:`support` section for more details. 121 122Style guide 123----------- 124 125We generally follow `pep8 126<https://www.python.org/dev/peps/pep-0008/>`_, with 120 columns 127instead of 79. We do *not* use form-feed (``^L``) characters to 128separate sections either. Compliance is tested automatically when 129you run the tests. 130 131Continuous Integration 132---------------------- 133 134All pull requests go through `GitHub Actions`_, which runs the tests on Linux 135and Mac OS X as well as the flake8 style checker. Windows builds run on AppVeyor_, 136while additional Unix-like platforms are tested on Golem_. 137 138.. _AppVeyor: https://ci.appveyor.com/project/borgbackup/borg/ 139.. _Golem: https://golem.enkore.de/view/Borg/ 140.. _GitHub Actions: https://github.com/borgbackup/borg/actions 141 142Output and Logging 143------------------ 144When writing logger calls, always use correct log level (debug only for 145debugging, info for informative messages, warning for warnings, error for 146errors, critical for critical errors/states). 147 148When directly talking to the user (e.g. Y/N questions), do not use logging, 149but directly output to stderr (not: stdout, it could be connected to a pipe). 150 151To control the amount and kinds of messages output emitted at info level, use 152flags like ``--stats`` or ``--list``, then create a topic logger for messages 153controlled by that flag. See ``_setup_implied_logging()`` in 154``borg/archiver.py`` for the entry point to topic logging. 155 156Building a development environment 157---------------------------------- 158 159First, just install borg into a virtual env :ref:`as described before <git-installation>`. 160 161To install some additional packages needed for running the tests, activate your 162virtual env and run:: 163 164 pip install -r requirements.d/development.txt 165 166 167Running the tests 168----------------- 169 170The tests are in the borg/testsuite package. 171 172To run all the tests, you need to have fakeroot installed. If you do not have 173fakeroot, you still will be able to run most tests, just leave away the 174`fakeroot -u` from the given command lines. 175 176To run the test suite use the following command:: 177 178 fakeroot -u tox # run all tests 179 180Some more advanced examples:: 181 182 # verify a changed tox.ini (run this after any change to tox.ini): 183 fakeroot -u tox --recreate 184 185 fakeroot -u tox -e py37 # run all tests, but only on python 3.7 186 187 fakeroot -u tox borg.testsuite.locking # only run 1 test module 188 189 fakeroot -u tox borg.testsuite.locking -- -k '"not Timer"' # exclude some tests 190 191 fakeroot -u tox borg.testsuite -- -v # verbose py.test 192 193Important notes: 194 195- When using ``--`` to give options to py.test, you MUST also give ``borg.testsuite[.module]``. 196 197 198Running more checks using coala 199------------------------------- 200 201First install coala and some checkers ("bears"): 202 203:: 204 205 pip install -r requirements.d/coala.txt 206 207You can now run coala from the toplevel directory; it will read its settings 208from ``.coafile`` there: 209 210:: 211 212 coala 213 214Some bears have additional requirements and they usually tell you about 215them in case they are missing. 216 217 218Adding a compression algorithm 219------------------------------ 220 221If you want to add a new compression algorithm, please refer to :issue:`1633` 222and leave a post there in order to discuss about the proposal. 223 224Documentation 225------------- 226 227Generated files 228~~~~~~~~~~~~~~~ 229 230Usage documentation (found in ``docs/usage/``) and man pages 231(``docs/man/``) are generated automatically from the command line 232parsers declared in the program and their documentation, which is 233embedded in the program (see archiver.py). These are committed to git 234for easier use by packagers downstream. 235 236When a command is added, a command line flag changed, added or removed, 237the usage docs need to be rebuilt as well:: 238 239 python setup.py build_usage 240 python setup.py build_man 241 242However, we prefer to do this as part of our :ref:`releasing` 243preparations, so it is generally not necessary to update these when 244submitting patches that change something about the command line. 245 246Building the docs with Sphinx 247~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 248 249The documentation (in reStructuredText format, .rst) is in docs/. 250 251To build the html version of it, you need to have Sphinx installed 252(in your Borg virtualenv with Python 3):: 253 254 pip install -r requirements.d/docs.txt 255 256Now run:: 257 258 cd docs/ 259 make html 260 261Then point a web browser at docs/_build/html/index.html. 262 263The website is updated automatically by ReadTheDocs through GitHub web hooks on the 264main repository. 265 266Using Vagrant 267------------- 268 269We use Vagrant for the automated creation of testing environments and borgbackup 270standalone binaries for various platforms. 271 272For better security, there is no automatic sync in the VM to host direction. 273The plugin `vagrant-scp` is useful to copy stuff from the VMs to the host. 274 275The "windows10" box requires the `reload` plugin (``vagrant plugin install vagrant-reload``). 276 277Usage:: 278 279 # To create and provision the VM: 280 vagrant up OS 281 # same, but use 6 VM cpus and 12 workers for pytest: 282 VMCPUS=6 XDISTN=12 vagrant up OS 283 # To create an ssh session to the VM: 284 vagrant ssh OS 285 # To execute a command via ssh in the VM: 286 vagrant ssh OS -c "command args" 287 # To shut down the VM: 288 vagrant halt OS 289 # To shut down and destroy the VM: 290 vagrant destroy OS 291 # To copy files from the VM (in this case, the generated binary): 292 vagrant scp OS:/vagrant/borg/borg.exe . 293 294 295Creating standalone binaries 296---------------------------- 297 298Make sure you have everything built and installed (including llfuse and fuse). 299When using the Vagrant VMs, pyinstaller will already be installed. 300 301With virtual env activated:: 302 303 pip install pyinstaller # or git checkout master 304 pyinstaller -F -n borg-PLATFORM borg/__main__.py 305 for file in dist/borg-*; do gpg --armor --detach-sign $file; done 306 307If you encounter issues, see also our `Vagrantfile` for details. 308 309.. note:: Standalone binaries built with pyinstaller are supposed to 310 work on same OS, same architecture (x86 32bit, amd64 64bit) 311 without external dependencies. 312 313 314.. _releasing: 315 316Creating a new release 317---------------------- 318 319Checklist: 320 321- Make sure all issues for this milestone are closed or moved to the 322 next milestone. 323- Check if there are any pending fixes for security issues. 324- Find and fix any low hanging fruit left on the issue tracker. 325- Check that GitHub Actions CI is happy. 326- Update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``. 327- Check version number of upcoming release in ``CHANGES.rst``. 328- Render ``CHANGES.rst`` via ``make html`` and check for markup errors. 329- Verify that ``MANIFEST.in`` and ``setup.py`` are complete. 330- ``python setup.py build_usage ; python setup.py build_man`` and 331 commit (be sure to build with Python 3.5 as Python 3.6 added `more 332 guaranteed hashing algorithms 333 <https://github.com/borgbackup/borg/issues/2123>`_). 334- Tag the release:: 335 336 git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z 337 338- Create a clean repo and use it for the following steps:: 339 340 git clone borg borg-clean 341 342 This makes sure no uncommitted files get into the release archive. 343 It will also reveal uncommitted required files. 344 Moreover, it makes sure the vagrant machines only get committed files and 345 do a fresh start based on that. 346- Run tox and/or binary builds on all supported platforms via vagrant, 347 check for test failures. 348- Create sdist, sign it, upload release to (test) PyPi: 349 350 :: 351 352 scripts/sdist-sign X.Y.Z 353 scripts/upload-pypi X.Y.Z test 354 scripts/upload-pypi X.Y.Z 355- Put binaries into dist/borg-OSNAME and sign them: 356 357 :: 358 359 scripts/sign-binaries 201912312359 360- Close the release milestone on GitHub. 361- `Update borgbackup.org 362 <https://github.com/borgbackup/borgbackup.github.io/pull/53/files>`_ with the 363 new version number and release date. 364- Announce on: 365 366 - Mailing list. 367 - Twitter. 368 - IRC channel (change ``/topic``). 369 370- Create a GitHub release, include: 371 372 * Standalone binaries (see above for how to create them). 373 374 + For OS X, document the OS X Fuse version in the README of the binaries. 375 OS X FUSE uses a kernel extension that needs to be compatible with the 376 code contained in the binary. 377 * A link to ``CHANGES.rst``. 378