1# Third-party libraries 2 3gRPC depends on several third-party libraries, their source code is available 4(usually as a git submodule) in this directory. 5 6## Guidelines on updating submodules 7 8- IMPORTANT: whenever possible, try to only update to a stable release of a library (= not to master / random commit). Depending on unreleased revisions 9 makes gRPC installation harder for users, as it forces them to always build the dependency from source and prevents them from using more 10 convenient installation channels (linux packages, package managers etc.) 11 12- bazel BUILD uses a different dependency model - whenever updating a submodule, also update the revision in `grpc_deps.bzl` so that bazel and 13 non-bazel builds stay in sync (this is actually enforced by a sanity check in some cases) 14 15## Considerations when adding a new third-party dependency 16 17- gRPC C++ needs to stay buildable/installable even if the submodules are not present (e.g. the tar.gz archive with gRPC doesn't contain the submodules), 18 assuming that the dependencies are already installed. This is a requirement for being able to provide a reasonable install process (e.g. using cmake) 19 and to support package managers for gRPC C++. 20 21- Adding a new dependency is a lot of work (both for us and for the users). 22 We currently support multiple build systems (BAZEL, cmake, make, ...) so adding a new dependency usually requires updates in multiple build systems 23 (often not trivial). The installation process also needs to continue to work (we do have distrib tests to test many of the possible installation scenarios, 24 but they are not perfect). Adding a new dependency also usually affects the installation instructions that need to be updated. 25 Also keep in mind that adding a new dependency can be quite disruptive 26 for the users and community - it means that all users will need to update their projects accordingly (for C++ projects often non-trivial) and 27 the community-provided C++ packages (e.g. vcpkg) will need to be updated as well. 28 29## Checklist for adding a new third-party dependency 30 31**READ THIS BEFORE YOU ADD A NEW DEPENDENCY** 32 33- [ ] Make sure you understand the hidden costs of adding a dependency (see section above) and that you understand the complexities of updating the build files. Maintenance of the build files isn't for free, so expect to be involved in maintenance tasks, cleanup and support (e.g resolving user bugs) of the build files in the future. 34 35- [ ] Once your change is ready, start an [adhoc run of artifact - packages - distribtests flow](https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod%3Agrpc%2Fcore%2Fexperimental%2Fgrpc_build_artifacts_multiplatform) and make sure everything passes (for technical reasons, not all the distribtests can run on each PR automatically). 36 37- [ ] Check the impact of the new dependency on the size of our distribution packages (compare BEFORE and AFTER) and post the comparison on your PR (it should not be approved without checking the impact sizes of packages first). The package sizes AFTER can be obtained from the adhoc package build from bullet point above. 38 39## Instructions for updating dependencies 40 41Usually the process is 42 431. update the submodule to selected commit (see guidance above) 442. update the dependency in `grpc_deps.bzl` to the same commit 453. update `tools/run_tests/sanity/check_submodules.sh` to make the sanity test pass 464. (when needed) run `tools/buildgen/generate_projects.sh` to regenerate the generated files 475. populate the bazel download mirror by running `bazel/update_mirror.sh` 48 49Updating some dependencies requires extra care. 50 51### Updating third_party/abseil-cpp 52 53- Two additional steps should be done before running `generate_projects.sh` above. 54 - Running `src/abseil-cpp/preprocessed_builds.yaml.gen.py`. 55 - Updating `abseil_version =` scripts in `templates/gRPC-C++.podspec.template` and 56 `templates/gRPC-Core.podspec.template`. 57- You can see an example of previous [upgrade](https://github.com/grpc/grpc/pull/24270). 58 59### Updating third_party/boringssl-with-bazel 60 61- Update the `third_party/boringssl-with-bazel` submodule to the latest [`main-with-bazel`](https://github.com/google/boringssl/tree/main-with-bazel) branch 62``` 63git submodule update --init # just to start in a clean state 64cd third_party/boringssl-with-bazel 65git fetch origin # fetch what's new in the boringssl repository 66git checkout origin/main-with-bazel # checkout the current state of main-with-bazel branch in the boringssl repo 67# Note the latest commit SHA on main-with-bazel branch 68cd ../.. # go back to grpc repo root 69git status # will show that there are new commits in third_party/boringssl-with-bazel 70git add third_party/boringssl-with-bazel # we actually want to update the changes to the submodule 71git commit -m "update submodule boringssl-with-bazel with origin/main-with-bazel" # commit 72``` 73 74- Update boringssl dependency in `bazel/grpc_deps.bzl` to the same commit SHA as main-with-bazel branch 75 - Update `http_archive(name = "boringssl",` section by updating the sha in `strip_prefix` and `urls` fields. 76 - Also, set `sha256` field to "" as the existing value is not valid. This will be added later once we know what that value is. 77 78- Update `tools/run_tests/sanity/check_submodules.sh` with the same commit 79 80- Commit these changes `git commit -m "update boringssl dependency to main-with-bazel commit SHA"` 81 82- Run `tools/buildgen/generate_projects.sh` to regenerate the generated files 83 - Because `sha256` in `bazel/grpc_deps.bzl` was left empty, you will get a DEBUG msg like this one: 84 ``` 85 Rule 'boringssl' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "SHA value" 86 ``` 87 - Commit the regenrated files `git commit -m "regenerate files"` 88 - Update `bazel/grpc_deps.bzl` with the SHA value shown in the above debug msg. Commit again `git commit -m "Updated sha256"` 89 90- Run `tools/distrib/generate_boringssl_prefix_header.sh` 91 - Commit again `git commit -m "generate boringssl prefix headers"` 92 93- Increment the boringssl podspec version number in 94 `templates/src/objective-c/BoringSSL-GRPC.podspec.template` and `templates/gRPC-Core.podspec.template`. 95 [example](https://github.com/grpc/grpc/pull/21527/commits/9d4411842f02f167209887f1f3d2b9ab5d14931a) 96 - Commit again `git commit -m "Increment podspec version"` 97 98- Run `tools/buildgen/generate_projects.sh` (yes, again) 99 - Commit again `git commit -m "Second regeneration"` 100 101- Create a PR with all the above commits. 102 103- Run `bazel/update_mirror.sh` to update GCS mirror. 104 105### Updating third_party/protobuf 106 107See http://go/grpc-third-party-protobuf-update-instructions (internal only) 108 109### Updating third_party/envoy-api 110 111Apart from the above steps, please perform the following two steps to generate the Python `xds-protos` package: 112 1131. Bump the version in the `tools/distrib/python/xds_protos/setup.py`; 1142. Run `tools/distrib/python/xds_protos/build_validate_upload.sh` to upload the built wheel. 115 116### Updating third_party/upb 117 118Since upb is vendored in the gRPC repo, you cannot use submodule to update it. Please follow the steps below. 119 1201. Update third_party/upb directory by running 121 `git subtree pull --squash --prefix=third_party/upb https://github.com/protocolbuffers/upb.git master` 1222. Update the dependency in `grpc_deps.bzl` to the same commit 1233. Populate the bazel download mirror by running `bazel/update_mirror.sh` 1244. Update `src/upb/gen_build_yaml.py` for newly added or removed upb files 1255. Run `tools/buildgen/generate_projects.sh` to regenerate the generated files 1266. Run `tools/codegen/core/gen_upb_api.sh` to regenerate upb files. 127 If you see breaking changes here, you may want to import upb into Google3 along with gRPC. 128 129### Updating third_party/xxhash 130 131TODO(https://github.com/Cyan4973/xxHash/issues/548): revisit LICENSE 132instructions if upstream splits library and CLI. 133 134The upstream xxhash repository contains a bunch of files that we don't want, so 135we employ a rather manual update flow to pick up just the bits we care about: 136 137``` 138git remote add xxhash https://github.com/Cyan4973/xxHash.git 139git fetch xxhash 140git show xxhash/dev:xxhash.h > third_party/xxhash/xxhash.h 141git show xxhash/dev:LICENSE | sed -nE '/^-+$/q;p' > third_party/xxhash/LICENSE 142``` 143