1# HOWTO: Setup your Development Environment 2 3## Linux 4 5Install the dependencies needed for your distribution. The top-level 6[README](../README.md) file should list the minimal development tools necessary 7to compile `google-cloud-cpp`. But for active development you may want to 8install additional tools to run the unit and integration tests. 9 10These instructions will describe how to install these tools for Ubuntu 18.04 11(Bionic Beaver). For other distributions you may consult the Dockerfile used by 12the integration tests. We use 13[Dockerfile.fedora-install](../ci/kokoro/docker/Dockerfile.fedora-install) to 14to enforce formatting for our builds. If you use a different distribution, you 15will need to use the corresponding package manager (`dnf`, `zypper`, `apk`, 16etc.) and find the corresponding package names. 17 18First, install the basic development tools: 19 20```console 21sudo apt update 22sudo apt install -y build-essential cmake git gcc g++ cmake \ 23 libc-ares-dev libc-ares2 libbenchmark-dev libcurl4-openssl-dev libssl-dev \ 24 make pkg-config tar wget zlib1g-dev 25``` 26 27Then install `clang-10` and some additional Clang tools that we use to enforce 28code style rules: 29 30```console 31sudo apt install -y clang-10 clang-tidy-10 clang-format-10 clang-tools-10 32sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-10 100 33sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-10 100 34``` 35 36Install buildifier tool, which we use to format `BUILD` files: 37 38```console 39sudo wget -q -O /usr/bin/buildifier https://github.com/bazelbuild/buildtools/releases/download/0.29.0/buildifier 40sudo chmod 755 /usr/bin/buildifier 41``` 42 43Install shfmt tool, which we use to format our shell scripts: 44 45```console 46curl -L -o /usr/local/bin/shfmt \ 47 "https://github.com/mvdan/sh/releases/download/v3.1.0/shfmt_v3.1.0_linux_amd64" 48chmod 755 /usr/local/bin/shfmt 49``` 50 51Install `cmake_format` to automatically format the CMake list files. We pin this 52tool to a specific version because the formatting changes when the "latest" 53version is updated, and we do not want the builds to break just 54because some third party changed something. 55 56```console 57sudo apt install -y python3 python3-pip 58pip3 install --upgrade pip3 59pip3 install cmake_format==0.6.8 60``` 61 62Install black, which we use to format our Python scripts: 63 64```console 65pip3 install black==19.3b0 66``` 67 68Install cspell for spell checking. 69 70```console 71sudo npm install -g cspell 72``` 73 74Install the Python modules used in the integration tests: 75 76```console 77pip3 install setuptools wheel 78pip3 install git+git://github.com/googleapis/python-storage@8cf6c62a96ba3fff7e5028d931231e28e5029f1c 79pip3 install flask==1.1.2 httpbin==0.7.0 scalpl==0.4.0 \ 80 crc32c==2.1 gunicorn==20.0.4 81``` 82 83Add the pip directory to your PATH: 84 85```console 86export PATH=$PATH:$HOME/.local/bin 87``` 88 89You need to install the Google Cloud SDK. These instructions work for a GCE 90VM, but you may need to adapt them for your environment. Check the instructions 91on the 92[Google Cloud SDK website](https://cloud.google.com/sdk/) for alternatives. 93 94```console 95./ci/install-cloud-sdk.sh 96``` 97 98### Clone and compile `google-cloud-cpp` 99 100You may need to clone and compile the code as described [here](setup-cmake-environment.md) 101 102Run the tests using: 103 104```console 105env -C cmake-out/home ctest --output-on-failure -LE integration-test 106``` 107 108Run the Google Cloud Storage integration tests: 109 110```console 111env -C cmake-out/home \ 112 $PWD/google/cloud/storage/ci/run_integration_tests_emulator_cmake.sh . 113``` 114 115Run the Google Cloud Bigtable integration tests: 116 117```console 118env -C cmake-out/home \ 119 $PWD/google/cloud/bigtable/ci/run_integration_tests_emulator_cmake.sh . 120``` 121 122### Installing Docker 123 124You may want to [install Docker](https://docs.docker.com/engine/installation/), 125this will allow you to use the build scripts to test on multiple platforms, 126as described in [CONTRIBUTING.md](../CONTRIBUTING.md). 127 128## Windows 129 130If you mainly use Windows as your development environment, you need to install 131a number of tools. We use [Chocolatey](https://www.chocolatey.com) to drive the 132installation, so you would need to install it first. This needs to be executed 133in a `cmd.exe` shell, running as the `Administrator`: 134 135```console 136> @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -ExecutionPolicy Bypass -Command "iex ( 137(New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" 138``` 139 140Then you can install the dependencies in the same shell: 141```console 142> choco install -y cmake git cmake.portable activeperl ninja golang yasm putty msys2 bazel 143> choco install -y visualstudio2019community visualstudio2019-workload-nativedesktop microsoft-build-tools 144``` 145 146### Connecting to GitHub with PuTTY 147 148This short recipe is offered to setup your SSH keys quickly using PuTTY. If 149you prefer another SSH client for Windows, please search the Internet for a 150tutorial on how to configure it. 151 152First, generate a private/public key pair with `puttygen`: 153 154```console 155> puttygen 156``` 157 158Then store the public key in your 159[GitHub Settings](https://github.com/settings/keys). 160 161Once you have generated the public/private key pair, start the SSH agent in the 162background: 163 164```console 165> pageant 166``` 167 168and use the menu to load the private key you generated above. Test the keys 169with: 170 171```console 172> plink -T git@github.com 173``` 174 175and do not forget to setup the `GIT_SSH` environment variable: 176 177```console 178> set GIT_SSH=plink 179``` 180 181### Clone `google-cloud-cpp` 182 183You may need to create a new key pair to connect to GitHub. Search the web 184for how to do this. Then you can clone the code: 185 186```console 187> cd \Users\%USERNAME% 188> git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git 189> cd google-cloud-cpp 190``` 191 192### Compile `google-cloud-cpp` using cmake and vcpkg 193 194### Download and compile `vcpkg` 195 196The previous installation should create a 197`Developer Command Prompt for VS 2017` entry in your "Windows" menu, use that 198entry to create a new shell. 199In that shell, install `vcpkg` the Microsoft-supported ports for many Open 200Source projects: 201 202```console 203> cd \Users\%USERNAME% 204> git clone --depth 10 https://github.com/Microsoft/vcpkg.git 205> cd vcpkg 206> .\bootstrap-vcpkg.bat 207``` 208 209You can get `vcpkg` to compile all the dependencies for `google-cloud-cpp` by 210installing `google-cloud-cpp` itself: 211 212```console 213> vcpkg.exe install google-cloud-cpp:x64-windows-static 214> vcpkg.exe integrate install 215``` 216 217Compile the code using: 218 219```console 220> cd \Users\%USERNAME%\google-cloud-cpp 221> call "c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" 222> cmake -GNinja -H. -Bcmake-out 223 -DCMAKE_BUILD_TYPE=Debug 224 -DCMAKE_TOOLCHAIN_FILE=C:\Users\%USERNAME%\vcpkg\scripts\buildsystems\vcpkg.cmake 225 -DVCPKG_TARGET_TRIPLET=x64-windows-static 226> cmake --build cmake-out 227``` 228 229Run the tests using: 230 231```console 232> cd cmake-out 233> ctest --output-on-failure 234``` 235 236### Compile `google-cloud-cpp` using bazel 237 238Due to Windows command line length limits, create an abbreviated output directory: 239```console 240mkdir c:\b 241``` 242 243Compile the code: 244```console 245cd \Users\%USERNAME%\google-cloud-cpp 246bazel --output_user_root="c:\b" build //google/cloud/...:all 247``` 248 249Run all the tests: 250```console 251bazel --output_user_root="c:\b" test //google/cloud/...:all 252``` 253 254## Appendix: Creating a Linux VM using Google Compute Engine 255 256From time to time you may want to setup a Linux VM in Google Compute Engine. 257This might be useful to run performance tests in isolation, but "close" to the 258service you are doing development for. The following instructions assume you 259have already created a project: 260 261```console 262$ PROJECT_ID=$(gcloud config get-value project) 263# Or manually set it if you have not configured your default project: 264``` 265 266Select a zone to run your VM: 267 268```console 269$ gcloud compute zones list 270$ ZONE=... # Pick a zone. 271``` 272 273Select the name of the VM: 274 275```console 276$ VM=... # e.g. VM=my-windows-devbox 277``` 278 279Then create the virtual machine using: 280 281```console 282# Googlers should consult go/drawfork before selecting an image. 283$ IMAGE_PROJECT=ubuntu-os-cloud 284$ IMAGE=$(gcloud compute images list \ 285 --project=${IMAGE_PROJECT} \ 286 --filter="family ~ ubuntu-1804" \ 287 --sort-by=~name \ 288 --format="value(name)" \ 289 --limit=1) 290$ PROJECT_NUMBER=$(gcloud projects list \ 291 --filter="project_id=${PROJECT_ID}" \ 292 --format="value(project_number)" \ 293 --limit=1) 294$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \ 295 --zone "${ZONE}" \ 296 --image "${IMAGE}" \ 297 --image-project "${IMAGE_PROJECT}" \ 298 --boot-disk-size "1024" --boot-disk-type "pd-standard" \ 299 --boot-disk-device-name "${VM}" \ 300 --service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ 301 --machine-type "n1-standard-8" \ 302 --subnet "default" \ 303 --maintenance-policy "MIGRATE" \ 304 --scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform" 305``` 306 307To login to this image use: 308 309```console 310$ gcloud compute ssh --ssh-flag=-A --zone=${ZONE} ${VM} 311``` 312 313## Appendix: Creating a Windows VM using Google Compute Engine 314 315If you do not have a Windows workstation, but need a Windows development 316environment to troubleshoot a test or build problem, it might be convenient to 317create a Windows VM. The following commands assume you have already created a 318project: 319 320```console 321$ PROJECT_ID=... # Set to your project id 322``` 323 324Select a zone to run your VM: 325 326```console 327$ gcloud compute zones list 328$ ZONE=... # Pick a zone close to where you are... 329``` 330 331Select the name of the VM: 332 333```console 334$ VM=... # e.g. VM=my-windows-devbox 335``` 336 337Then create the virtual machine using: 338 339```console 340$ IMAGE=$(gcloud compute images list \ 341 --sort-by=~name \ 342 --format="value(name)" \ 343 --limit=1) 344$ PROJECT_NUMBER=$(gcloud projects list \ 345 --filter="project_id=${PROJECT_ID}" \ 346 --format="value(project_number)" \ 347 --limit=1) 348$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \ 349 --zone "${ZONE}" \ 350 --image "${IMAGE}" --image-project "windows-cloud" \ 351 --boot-disk-size "1024" --boot-disk-type "pd-standard" \ 352 --boot-disk-device-name "${VM}" \ 353 --service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ 354 --machine-type "n1-standard-8" \ 355 --subnet "default" \ 356 --maintenance-policy "MIGRATE" \ 357 --scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform" 358``` 359 360Reset the password for your account: 361 362```console 363$ gcloud compute --project "${PROJECT_ID}" reset-windows-password --zone "${ZONE}" "${VM}" 364``` 365 366Save that password in some kind of password manager. Then connect to the VM 367using your favorite RDP client. The Google Cloud Compute Engine 368[documentation](https://cloud.google.com/compute/docs/quickstart-windows) 369suggests some third-party clients that may be useful. 370