• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..31-Mar-2022-

converter.pyH A D31-Mar-202216.8 KiB482361

read-json.pyH A D31-Mar-20221.3 KiB4331

readme.mdH A D31-Mar-20225.2 KiB10788

sync-to-github.shH A D31-Mar-20224.9 KiB160111

readme.md

1# Github synchronization scripts
2
3This tool aims to help synchronizing changes from mozilla-central to Github on pushes.
4This is useful for Gecko sub-projects that have Github mirrors, like `gfx/wr` linking to `https://github.com/servo/webrender`.
5Originally, the tools were developed in `https://github.com/staktrace/wrupdater`,
6then got moved under `gfx/wr/ci-scripts/wrupdater`,
7and finally migrated here while also abstracting away from WebRender specifically.
8
9The main entry point is the `sync-to-github.sh` script that is called with the following arguments:
10  1. name of the project, matching the repository under `https://github.com/moz-gfx` user (e.g. `webrender`)
11  2. relative folder in mozilla-central, which is the upstream for the changes (e.g. `gfx/wr`)
12  3. downstream repository specified as "organization/project-name" (e.g. `servo/webrender`)
13  4. name to call for auto-approving the pull request (e.g. `bors` or `@bors-servo`)
14
15It creates a staging directory at `~/.ghsync` if one doesn't already exist,
16and clones the the downstream repo into it.
17The script also requires the `GECKO_PATH` environment variable
18to point to a mercurial clone of `mozilla-central`, and access to the
19taskcluster secrets service to get a Github API token.
20
21The `sync-to-github.sh` script does some setup steps but the bulk of the actual work
22is done by the `converter.py` script. This script scans the mercurial
23repository for new changes to the relative folder in m-c,
24and adds commits to the git repository corresponding to those changes.
25There are some details in the implementation that make it more robust
26than simply exporting patches and attempting to reapply them;
27in particular it builds a commit tree structure that mirrors what is found in
28the `mozilla-central` repository with respect to branches and merges.
29So if conflicting changes land on autoland and inbound, and then get
30merged, the git repository commits will have the same structure with
31a fork/merge in the commit history. This was discovered to be
32necessary after a previous version ran into multiple cases where
33the simple patch approach didn't really work.
34
35One of the actions the `converter.py` takes is to find the last sync point
36between Github and mozilla-central. This is done based on the following markers:
37  - commit message containing the string "[ghsync] From https://hg.mozilla.org/mozilla-central/rev/xxx"
38  - commit message containing the string "[wrupdater] From https://hg.mozilla.org/mozilla-central/rev/xxx"
39  - commit with tag "mozilla-xxx"
40(where xxx is always a mozilla-central hg revision identifier).
41
42Once the converter is done converting, the `sync-to-github.sh` script
43finishes the process by pushing the new commits to the `github-sync` branch
44of the `https://github.com/moz-gfx/<project-name>` repository,
45and generating a pull request against the downstream repository. It also
46leaves a comment on the PR that triggers testing and automatic merge of the PR.
47If there is already a pull request (perhaps from a previous run) the
48pre-existing PR is force-updated instead. This allows for graceful
49handling of scenarios where the PR failed to get merged (e.g. due to
50CI failures on the Github side).
51
52The script is intended to by run by taskcluster for any changes that
53touch the relative folder that land on `mozilla-central`. This may mean
54that multiple instances of this script run concurrently, or even out
55of order (i.e. the task for an older m-c push runs after the task for
56a newer m-c push). The script was written with these possibilities in
57mind and should be able to eventually recover from any such scenario
58automatically (although it may take additional changes to mozilla-central
59for such recovery to occur). That being said, the number of pathological
60scenarios here is quite large and they were not really tested.
61
62## Ownership and access
63
64When this tool is run in Firefox CI, it needs to have push permissions to
65the `moz-gfx` github user's account. It gets this permission via a secret token
66stored in the Firefox CI taskcluster secrets service. If you need to update
67the token, you need to find somebody who is a member of the
68[webrender-ci access group](https://people.mozilla.org/a/webrender-ci/). The
69Google Drive associated with that access group has additional documentation
70on the `moz-gfx` github user and the secret token.
71
72## Debugging
73
74To debug the converter.py script, you need to have a hg checkout of
75mozilla-central, let's assume it's at $MOZILLA. First create a virtualenv
76with the right dependencies installed:
77
78```
79mkdir -p $HOME/.ghsync
80virtualenv --python=python3 $HOME/.ghsync/venv
81source $HOME/.ghsync/venv/bin/activate
82pip3 install -r $MOZILLA/taskcluster/docker/github-sync/requirements.txt
83```
84
85Also create a checkout of the downstream github repo and set up a `github-sync`
86branch to the point where you want port commits to. For example, for WebRender
87you'd do:
88
89```
90cd $HOME/.ghsync
91git clone https://github.com/servo/webrender
92cd webrender
93git checkout -b github-sync master
94```
95
96(You can set the github-sync branch to a past revision if you want to replicate
97a failure that already got committed).
98
99Then run the converter from your hg checkout:
100
101```
102cd $MOZILLA
103tools/github-sync/converter.py $HOME/.ghsync/webrender gfx/wr
104```
105
106You can set the DEBUG variable in the script to True to get more output.
107