1# HOWTO: Setup your Development Environment 2 3## Linux 4 5Install the dependencies needed for your distribution. The top-level 6[README](../README.md) file should list the minimal development tools necessary 7to compile `google-cloud-cpp`. But for active development you may want to 8install additional tools to run the unit and integration tests. 9 10These instructions will describe how to install these tools for 11Ubuntu 18.04 (Bionic Beaver). For other distributions you may consult the 12Dockerfile used by the integration tests. For example, 13[Dockerfile.ubuntu](../ci/kokoro/docker/Dockerfile.ubuntu), or 14[Dockerfile.fedora](../ci/kokoro/docker/Dockerfile.fedora). 15 16First, install the basic development tools: 17 18```console 19sudo apt update 20sudo apt install -y build-essential cmake git gcc g++ cmake \ 21 libc-ares-dev libc-ares2 libcurl4-openssl-dev libssl-dev make \ 22 pkg-config tar wget zlib1g-dev 23``` 24 25Then install `clang-6.0` and some additional Clang tools that we use to enforce 26code style rules: 27 28```console 29sudo apt install -y clang-6.0 clang-tidy clang-format-7 clang-tools 30sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-6.0 100 31sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-7 100 32``` 33 34Install buildifier tool, which we use to format `BUILD` files: 35 36```console 37sudo wget -q -O /usr/bin/buildifier https://github.com/bazelbuild/buildtools/releases/download/0.17.2/buildifier 38sudo chmod 755 /usr/bin/buildifier 39``` 40 41Install cmake_format to automatically format the CMake list files. We pin this 42tool to a specific version because the formatting changes when the "latest" 43version is updated, and we do not want the builds to break just 44because some third party changed something. 45 46```console 47sudo apt install -y python3 python3-pip 48pip3 install --upgrade pip3 49pip3 install cmake_format==0.6.0 50``` 51 52Install the Python modules used in the integration tests: 53 54```console 55pip3 install flask==1.1.1 Werkzeug==1.0.0 httpbin==0.7.0 \ 56 gevent==1.4.0 gunicorn==19.10.0 crc32c==2.0 57``` 58 59Add the pip directory to your PATH: 60 61```console 62export PATH=$PATH:$HOME/.local/bin 63``` 64 65You need to install the Google Cloud SDK. These instructions work for a GCE 66VM, but you may need to adapt them for your environment. Check the instructions 67on the 68[Google Cloud SDK website](https://cloud.google.com/sdk/) for alternatives. 69 70```console 71./ci/install-cloud-sdk.sh 72``` 73 74### Clone and compile `google-cloud-cpp` 75 76You may need to clone and compile the code as described [here](setup-cmake-environment.md) 77 78Run the tests using: 79 80```console 81env -C cmake-out/home ctest --output-on-failure -LE integration-test 82``` 83 84Run the Google Cloud Storage integration tests: 85 86```console 87env -C cmake-out/home \ 88 $PWD/google/cloud/storage/ci/run_integration_tests_emulator_cmake.sh 89``` 90 91Run the Google Cloud Bigtable integration tests: 92 93```console 94env -C cmake-out/home \ 95 $PWD/google/cloud/bigtable/ci/run_integration_tests_emulator_cmake.sh 96``` 97 98### Installing Docker 99 100You may want to [install Docker](https://docs.docker.com/engine/installation/), 101this will allow you to use the build scripts to test on multiple platforms, 102as described in [CONTRIBUTING.md](../CONTRIBUTING.md). 103 104## Windows 105 106If you mainly use Windows as your development environment, you need to install 107a number of tools. We use [Chocolatey](https://www.chocolatey.com) to drive the 108installation, so you would need to install it first. This needs to be executed 109in a `cmd.exe` shell, running as the `Administrator`: 110 111```console 112> @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -ExecutionPolicy Bypass -Command "iex ( 113(New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" 114``` 115 116Then you can install the dependencies in the same shell: 117```console 118> choco install -y cmake git cmake.portable activeperl ninja golang yasm putty msys2 bazel 119> choco install -y visualstudio2019community visualstudio2019-workload-nativedesktop microsoft-build-tools 120``` 121 122### Connecting to GitHub with PuTTY 123 124This short recipe is offered to setup your SSH keys quickly using PuTTY. If 125you prefer another SSH client for Windows, please search the Internet for a 126tutorial on how to configure it. 127 128First, generate a private/public key pair with `puttygen`: 129 130```console 131> puttygen 132``` 133 134Then store the public key in your 135[GitHub Settings](https://github.com/settings/keys). 136 137Once you have generated the public/private key pair, start the SSH agent in the 138background: 139 140```console 141> pageant 142``` 143 144and use the menu to load the private key you generated above. Test the keys 145with: 146 147```console 148> plink -T git@github.com 149``` 150 151and do not forget to setup the `GIT_SSH` environment variable: 152 153```console 154> set GIT_SSH=plink 155``` 156 157### Clone `google-cloud-cpp` 158 159You may need to create a new key pair to connect to GitHub. Search the web 160for how to do this. Then you can clone the code: 161 162```console 163> cd \Users\%USERNAME% 164> git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git 165> cd google-cloud-cpp 166``` 167 168### Compile `google-cloud-cpp` using cmake and vcpkg 169 170### Download and compile `vcpkg` 171 172The previous installation should create a 173`Developer Command Prompt for VS 2017` entry in your "Windows" menu, use that 174entry to create a new shell. 175In that shell, install `vcpkg` the Microsoft-supported ports for many Open 176Source projects: 177 178```console 179> cd \Users\%USERNAME% 180> git clone --depth 10 https://github.com/Microsoft/vcpkg.git 181> cd vcpkg 182> .\bootstrap-vcpkg.bat 183``` 184 185You can get `vcpkg` to compile all the dependencies for `google-cloud-cpp` by 186installing `google-cloud-cpp` itself: 187 188```console 189> vcpkg.exe install google-cloud-cpp:x64-windows-static 190> vcpkg.exe integrate install 191``` 192 193Compile the code using: 194 195```console 196> cd \Users\%USERNAME%\google-cloud-cpp 197> call "c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" 198> cmake -GNinja -H. -Bcmake-out 199 -DCMAKE_BUILD_TYPE=Debug 200 -DCMAKE_TOOLCHAIN_FILE=C:\Users\%USERNAME%\vcpkg\scripts\buildsystems\vcpkg.cmake 201 -DVCPKG_TARGET_TRIPLET=x64-windows-static 202> cmake --build cmake-out 203``` 204 205Run the tests using: 206 207```console 208> cd cmake-out 209> ctest --output-on-failure 210``` 211 212### Compile `google-cloud-cpp` using bazel 213 214Due to Windows command line length limits, create an abbreviated output directory: 215```console 216mkdir c:\b 217``` 218 219Compile the code: 220```console 221cd \Users\%USERNAME%\google-cloud-cpp 222bazel --output_user_root="c:\b" build //google/cloud/...:all 223``` 224 225Run all the tests: 226```console 227bazel --output_user_root="c:\b" test //google/cloud/...:all 228``` 229 230## Appendix: Creating a Linux VM using Google Compute Engine 231 232From time to time you may want to setup a Linux VM in Google Compute Engine. 233This might be useful to run performance tests in isolation, but "close" to the 234service you are doing development for. The following instructions assume you 235have already created a project: 236 237```console 238$ PROJECT_ID=$(gcloud config get-value project) 239# Or manually set it if you have not configured your default project: 240``` 241 242Select a zone to run your VM: 243 244```console 245$ gcloud compute zones list 246$ ZONE=... # Pick a zone. 247``` 248 249Select the name of the VM: 250 251```console 252$ VM=... # e.g. VM=my-windows-devbox 253``` 254 255Then create the virtual machine using: 256 257```console 258# Googlers should consult go/drawfork before selecting an image. 259$ IMAGE_PROJECT=ubuntu-os-cloud 260$ IMAGE=$(gcloud compute images list \ 261 --project=${IMAGE_PROJECT} \ 262 --filter="family ~ ubuntu-1804" \ 263 --sort-by=~name \ 264 --format="value(name)" \ 265 --limit=1) 266$ PROJECT_NUMBER=$(gcloud projects list \ 267 --filter="project_id=${PROJECT_ID}" \ 268 --format="value(project_number)" \ 269 --limit=1) 270$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \ 271 --zone "${ZONE}" \ 272 --image "${IMAGE}" \ 273 --image-project "${IMAGE_PROJECT}" \ 274 --boot-disk-size "1024" --boot-disk-type "pd-standard" \ 275 --boot-disk-device-name "${VM}" \ 276 --service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ 277 --machine-type "n1-standard-8" \ 278 --subnet "default" \ 279 --maintenance-policy "MIGRATE" \ 280 --scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform" 281``` 282 283To login to this image use: 284 285```console 286$ gcloud compute ssh --ssh-flag=-A --zone=${ZONE} ${VM} 287``` 288 289## Appendix: Creating a Windows VM using Google Compute Engine 290 291If you do not have a Windows workstation, but need a Windows development 292environment to troubleshoot a test or build problem, it might be convenient to 293create a Windows VM. The following commands assume you have already created a 294project: 295 296```console 297$ PROJECT_ID=... # Set to your project id 298``` 299 300Select a zone to run your VM: 301 302```console 303$ gcloud compute zones list 304$ ZONE=... # Pick a zone close to where you are... 305``` 306 307Select the name of the VM: 308 309```console 310$ VM=... # e.g. VM=my-windows-devbox 311``` 312 313Then create the virtual machine using: 314 315```console 316$ IMAGE=$(gcloud compute images list \ 317 --sort-by=~name \ 318 --format="value(name)" \ 319 --limit=1) 320$ PROJECT_NUMBER=$(gcloud projects list \ 321 --filter="project_id=${PROJECT_ID}" \ 322 --format="value(project_number)" \ 323 --limit=1) 324$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \ 325 --zone "${ZONE}" \ 326 --image "${IMAGE}" --image-project "windows-cloud" \ 327 --boot-disk-size "1024" --boot-disk-type "pd-standard" \ 328 --boot-disk-device-name "${VM}" \ 329 --service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ 330 --machine-type "n1-standard-8" \ 331 --subnet "default" \ 332 --maintenance-policy "MIGRATE" \ 333 --scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform" 334``` 335 336Reset the password for your account: 337 338```console 339$ gcloud compute --project "${PROJECT_ID}" reset-windows-password --zone "${ZONE}" "${VM}" 340``` 341 342Save that password in some kind of password manager. Then connect to the VM 343using your favorite RDP client. The Google Cloud Compute Engine 344[documentation](https://cloud.google.com/compute/docs/quickstart-windows) 345suggests some third-party clients that may be useful. 346