1# Github synchronization scripts 2 3This tool aims to help synchronizing changes from mozilla-central to Github on pushes. 4This is useful for Gecko sub-projects that have Github mirrors, like `gfx/wr` linking to `https://github.com/servo/webrender`. 5Originally, the tools were developed in `https://github.com/staktrace/wrupdater`, 6then got moved under `gfx/wr/ci-scripts/wrupdater`, 7and finally migrated here while also abstracting away from WebRender specifically. 8 9The main entry point is the `sync-to-github.sh` script that is called with the following arguments: 10 1. name of the project, matching the repository under `https://github.com/moz-gfx` user (e.g. `webrender`) 11 2. relative folder in mozilla-central, which is the upstream for the changes (e.g. `gfx/wr`) 12 3. downstream repository specified as "organization/project-name" (e.g. `servo/webrender`) 13 4. name to call for auto-approving the pull request (e.g. `bors` or `@bors-servo`) 14 15It creates a staging directory at `~/.ghsync` if one doesn't already exist, 16and clones the the downstream repo into it. 17The script also requires the `GECKO_PATH` environment variable 18to point to a mercurial clone of `mozilla-central`, and access to the 19taskcluster secrets service to get a Github API token. 20 21The `sync-to-github.sh` script does some setup steps but the bulk of the actual work 22is done by the `converter.py` script. This script scans the mercurial 23repository for new changes to the relative folder in m-c, 24and adds commits to the git repository corresponding to those changes. 25There are some details in the implementation that make it more robust 26than simply exporting patches and attempting to reapply them; 27in particular it builds a commit tree structure that mirrors what is found in 28the `mozilla-central` repository with respect to branches and merges. 29So if conflicting changes land on autoland and inbound, and then get 30merged, the git repository commits will have the same structure with 31a fork/merge in the commit history. This was discovered to be 32necessary after a previous version ran into multiple cases where 33the simple patch approach didn't really work. 34 35One of the actions the `converter.py` takes is to find the last sync point 36between Github and mozilla-central. This is done based on the following markers: 37 - commit message containing the string "[ghsync] From https://hg.mozilla.org/mozilla-central/rev/xxx" 38 - commit message containing the string "[wrupdater] From https://hg.mozilla.org/mozilla-central/rev/xxx" 39 - commit with tag "mozilla-xxx" 40(where xxx is always a mozilla-central hg revision identifier). 41 42Once the converter is done converting, the `sync-to-github.sh` script 43finishes the process by pushing the new commits to the `github-sync` branch 44of the `https://github.com/moz-gfx/<project-name>` repository, 45and generating a pull request against the downstream repository. It also 46leaves a comment on the PR that triggers testing and automatic merge of the PR. 47If there is already a pull request (perhaps from a previous run) the 48pre-existing PR is force-updated instead. This allows for graceful 49handling of scenarios where the PR failed to get merged (e.g. due to 50CI failures on the Github side). 51 52The script is intended to by run by taskcluster for any changes that 53touch the relative folder that land on `mozilla-central`. This may mean 54that multiple instances of this script run concurrently, or even out 55of order (i.e. the task for an older m-c push runs after the task for 56a newer m-c push). The script was written with these possibilities in 57mind and should be able to eventually recover from any such scenario 58automatically (although it may take additional changes to mozilla-central 59for such recovery to occur). That being said, the number of pathological 60scenarios here is quite large and they were not really tested. 61 62## Ownership and access 63 64When this tool is run in Firefox CI, it needs to have push permissions to 65the `moz-gfx` github user's account. It gets this permission via a secret token 66stored in the Firefox CI taskcluster secrets service. If you need to update 67the token, you need to find somebody who is a member of the 68[webrender-ci access group](https://people.mozilla.org/a/webrender-ci/). The 69Google Drive associated with that access group has additional documentation 70on the `moz-gfx` github user and the secret token. 71 72## Debugging 73 74To debug the converter.py script, you need to have a hg checkout of 75mozilla-central, let's assume it's at $MOZILLA. First create a virtualenv 76with the right dependencies installed: 77 78``` 79mkdir -p $HOME/.ghsync 80virtualenv --python=python3 $HOME/.ghsync/venv 81source $HOME/.ghsync/venv/bin/activate 82pip3 install -r $MOZILLA/taskcluster/docker/github-sync/requirements.txt 83``` 84 85Also create a checkout of the downstream github repo and set up a `github-sync` 86branch to the point where you want port commits to. For example, for WebRender 87you'd do: 88 89``` 90cd $HOME/.ghsync 91git clone https://github.com/servo/webrender 92cd webrender 93git checkout -b github-sync master 94``` 95 96(You can set the github-sync branch to a past revision if you want to replicate 97a failure that already got committed). 98 99Then run the converter from your hg checkout: 100 101``` 102cd $MOZILLA 103tools/github-sync/converter.py $HOME/.ghsync/webrender gfx/wr 104``` 105 106You can set the DEBUG variable in the script to True to get more output. 107