1# AV1 Codec Library 2 3## Contents 41. [Building the lib and applications](#building-the-library-and-applications) 5 - [Prerequisites](#prerequisites) 6 - [Get the code](#get-the-code) 7 - [Basics](#basic-build) 8 - [Configuration options](#configuration-options) 9 - [Dylib builds](#dylib-builds) 10 - [Debugging](#debugging) 11 - [Cross compiling](#cross-compiling) 12 - [Sanitizer support](#sanitizers) 13 - [MSVC builds](#microsoft-visual-studio-builds) 14 - [Xcode builds](#xcode-builds) 15 - [Emscripten builds](#emscripten-builds) 16 - [Extra Build Flags](#extra-build-flags) 172. [Testing the library](#testing-the-av1-codec) 18 - [Basics](#testing-basics) 19 - [Unit tests](#1_unit-tests) 20 - [Example tests](#2_example-tests) 21 - [Encoder tests](#3_encoder-tests) 22 - [IDE hosted tests](#ide-hosted-tests) 23 - [Downloading test data](#downloading-the-test-data) 24 - [Adding a new test data file](#adding-a-new-test-data-file) 25 - [Additional test data](#additional-test-data) 26 - [Sharded testing](#sharded-testing) 27 - [Running tests directly](#1_running-test_libaom-directly) 28 - [Running tests via CMake](#2_running-the-tests-via-the-cmake-build) 293. [Coding style](#coding-style) 304. [Submitting patches](#submitting-patches) 31 - [Login cookie](#login-cookie) 32 - [Contributor agreement](#contributor-agreement) 33 - [Testing your code](#testing-your-code) 34 - [Commit message hook](#commit-message-hook) 35 - [Upload your change](#upload-your-change) 36 - [Incorporating Reviewer Comments](#incorporating-reviewer-comments) 37 - [Submitting your change](#submitting-your-change) 38 - [Viewing change status](#viewing-the-status-of-uploaded-changes) 395. [Support](#support) 406. [Bug reports](#bug-reports) 41 42## Building the library and applications 43 44### Prerequisites 45 46 1. [CMake](https://cmake.org) version 3.5 or higher. 47 2. [Git](https://git-scm.com/). 48 3. [Perl](https://www.perl.org/). 49 4. For x86 targets, [yasm](http://yasm.tortall.net/), which is preferred, or a 50 recent version of [nasm](http://www.nasm.us/). 51 5. Building the documentation requires [doxygen](http://doxygen.org). 52 6. Building the unit tests requires [Python](https://www.python.org/). 53 7. Emscripten builds require the portable 54 [EMSDK](https://kripken.github.io/emscripten-site/index.html). 55 56### Get the code 57 58The AV1 library source code is stored in the Alliance for Open Media Git 59repository: 60 61~~~ 62 $ git clone https://aomedia.googlesource.com/aom 63 # By default, the above command stores the source in the aom directory: 64 $ cd aom 65~~~ 66 67### Basic build 68 69CMake replaces the configure step typical of many projects. Running CMake will 70produce configuration and build files for the currently selected CMake 71generator. For most systems the default generator is Unix Makefiles. The basic 72form of a makefile build is the following: 73 74~~~ 75 $ cmake path/to/aom 76 $ make 77~~~ 78 79The above will generate a makefile build that produces the AV1 library and 80applications for the current host system after the make step completes 81successfully. The compiler chosen varies by host platform, but a general rule 82applies: On systems where cc and c++ are present in $PATH at the time CMake is 83run the generated build will use cc and c++ by default. 84 85### Configuration options 86 87The AV1 codec library has a great many configuration options. These come in two 88varieties: 89 90 1. Build system configuration options. These have the form `ENABLE_FEATURE`. 91 2. AV1 codec configuration options. These have the form `CONFIG_FEATURE`. 92 93Both types of options are set at the time CMake is run. The following example 94enables ccache and disables the AV1 encoder: 95 96~~~ 97 $ cmake path/to/aom -DENABLE_CCACHE=1 -DCONFIG_AV1_ENCODER=0 98 $ make 99~~~ 100 101The available configuration options are too numerous to list here. Build system 102configuration options can be found at the top of the CMakeLists.txt file found 103in the root of the AV1 repository, and AV1 codec configuration options can 104currently be found in the file `build/cmake/aom_config_defaults.cmake`. 105 106### Dylib builds 107 108A dylib (shared object) build of the AV1 codec library can be enabled via the 109CMake built in variable `BUILD_SHARED_LIBS`: 110 111~~~ 112 $ cmake path/to/aom -DBUILD_SHARED_LIBS=1 113 $ make 114~~~ 115 116This is currently only supported on non-Windows targets. 117 118### Debugging 119 120Depending on the generator used there are multiple ways of going about 121debugging AV1 components. For single configuration generators like the Unix 122Makefiles generator, setting `CMAKE_BUILD_TYPE` to Debug is sufficient: 123 124~~~ 125 $ cmake path/to/aom -DCMAKE_BUILD_TYPE=Debug 126~~~ 127 128For Xcode, mainly because configuration controls for Xcode builds are buried two 129configuration windows deep and must be set for each subproject within the Xcode 130IDE individually, `CMAKE_CONFIGURATION_TYPES` should be set to Debug: 131 132~~~ 133 $ cmake path/to/aom -G Xcode -DCMAKE_CONFIGURATION_TYPES=Debug 134~~~ 135 136For Visual Studio the in-IDE configuration controls should be used. Simply set 137the IDE project configuration to Debug to allow for stepping through the code. 138 139In addition to the above it can sometimes be useful to debug only C and C++ 140code. To disable all assembly code and intrinsics set `AOM_TARGET_CPU` to 141generic at generation time: 142 143~~~ 144 $ cmake path/to/aom -DAOM_TARGET_CPU=generic 145~~~ 146 147### Cross compiling 148 149For the purposes of building the AV1 codec and applications and relative to the 150scope of this guide, all builds for architectures differing from the native host 151architecture will be considered cross compiles. The AV1 CMake build handles 152cross compiling via the use of toolchain files included in the AV1 repository. 153The toolchain files available at the time of this writing are: 154 155 - arm64-ios.cmake 156 - arm64-linux-gcc.cmake 157 - arm64-mingw-gcc.cmake 158 - armv7-ios.cmake 159 - armv7-linux-gcc.cmake 160 - armv7-mingw-gcc.cmake 161 - armv7s-ios.cmake 162 - mips32-linux-gcc.cmake 163 - mips64-linux-gcc.cmake 164 - x86-ios-simulator.cmake 165 - x86-linux.cmake 166 - x86-macos.cmake 167 - x86-mingw-gcc.cmake 168 - x86\_64-ios-simulator.cmake 169 - x86\_64-mingw-gcc.cmake 170 171The following example demonstrates use of the x86-macos.cmake toolchain file on 172a x86\_64 MacOS host: 173 174~~~ 175 $ cmake path/to/aom \ 176 -DCMAKE_TOOLCHAIN_FILE=path/to/aom/build/cmake/toolchains/x86-macos.cmake 177 $ make 178~~~ 179 180To build for an unlisted target creation of a new toolchain file is the best 181solution. The existing toolchain files can be used a starting point for a new 182toolchain file since each one exposes the basic requirements for toolchain files 183as used in the AV1 codec build. 184 185As a temporary work around an unoptimized AV1 configuration that builds only C 186and C++ sources can be produced using the following commands: 187 188~~~ 189 $ cmake path/to/aom -DAOM_TARGET_CPU=generic 190 $ make 191~~~ 192 193In addition to the above it's important to note that the toolchain files 194suffixed with gcc behave differently than the others. These toolchain files 195attempt to obey the $CROSS environment variable. 196 197### Sanitizers 198 199Sanitizer integration is built-in to the CMake build system. To enable a 200sanitizer, add `-DSANITIZE=<type>` to the CMake command line. For example, to 201enable address sanitizer: 202 203~~~ 204 $ cmake path/to/aom -DSANITIZE=address 205 $ make 206~~~ 207 208Sanitizers available vary by platform, target, and compiler. Consult your 209compiler documentation to determine which, if any, are available. 210 211### Microsoft Visual Studio builds 212 213Building the AV1 codec library in Microsoft Visual Studio is supported. The 214following example demonstrates generating projects and a solution for the 215Microsoft IDE: 216 217~~~ 218 # This does not require a bash shell; command.exe is fine. 219 $ cmake path/to/aom -G "Visual Studio 15 2017" 220~~~ 221 222### Xcode builds 223 224Building the AV1 codec library in Xcode is supported. The following example 225demonstrates generating an Xcode project: 226 227~~~ 228 $ cmake path/to/aom -G Xcode 229~~~ 230 231### Emscripten builds 232 233Building the AV1 codec library with Emscripten is supported. Typically this is 234used to hook into the AOMAnalyzer GUI application. These instructions focus on 235using the inspector with AOMAnalyzer, but all tools can be built with 236Emscripten. 237 238It is assumed here that you have already downloaded and installed the EMSDK, 239installed and activated at least one toolchain, and setup your environment 240appropriately using the emsdk\_env script. 241 2421. Download [AOMAnalyzer](https://people.xiph.org/~mbebenita/analyzer/). 243 2442. Configure the build: 245 246~~~ 247 $ cmake path/to/aom \ 248 -DENABLE_CCACHE=1 \ 249 -DAOM_TARGET_CPU=generic \ 250 -DENABLE_DOCS=0 \ 251 -DENABLE_TESTS=0 \ 252 -DCONFIG_ACCOUNTING=1 \ 253 -DCONFIG_INSPECTION=1 \ 254 -DCONFIG_MULTITHREAD=0 \ 255 -DCONFIG_RUNTIME_CPU_DETECT=0 \ 256 -DCONFIG_WEBM_IO=0 \ 257 -DCMAKE_TOOLCHAIN_FILE=path/to/emsdk-portable/.../Emscripten.cmake 258~~~ 259 2603. Build it: run make if that's your generator of choice: 261 262~~~ 263 $ make inspect 264~~~ 265 2664. Run the analyzer: 267 268~~~ 269 # inspect.js is in the examples sub directory of the directory in which you 270 # executed cmake. 271 $ path/to/AOMAnalyzer path/to/examples/inspect.js path/to/av1/input/file 272~~~ 273 274### Extra build flags 275 276Three variables allow for passing of additional flags to the build system. 277 278- AOM\_EXTRA\_C\_FLAGS 279- AOM\_EXTRA\_CXX\_FLAGS 280- AOM\_EXTRA\_EXE\_LINKER\_FLAGS 281 282The build system attempts to ensure the flags passed through the above variables 283are passed to tools last in order to allow for override of default behavior. 284These flags can be used, for example, to enable asserts in a release build: 285 286~~~ 287 $ cmake path/to/aom \ 288 -DCMAKE_BUILD_TYPE=Release \ 289 -DAOM_EXTRA_C_FLAGS=-UNDEBUG \ 290 -DAOM_EXTRA_CXX_FLAGS=-UNDEBUG 291~~~ 292 293## Testing the AV1 codec 294 295### Testing basics 296 297There are several methods of testing the AV1 codec. All of these methods require 298the presence of the AV1 source code and a working build of the AV1 library and 299applications. 300 301#### 1. Unit tests: 302 303The unit tests can be run at build time: 304 305~~~ 306 # Before running the make command the LIBAOM_TEST_DATA_PATH environment 307 # variable should be set to avoid downloading the test files to the 308 # cmake build configuration directory. 309 $ cmake path/to/aom 310 # Note: The AV1 CMake build creates many test targets. Running make 311 # with multiple jobs will speed up the test run significantly. 312 $ make runtests 313~~~ 314 315#### 2. Example tests: 316 317The example tests require a bash shell and can be run in the following manner: 318 319~~~ 320 # See the note above about LIBAOM_TEST_DATA_PATH above. 321 $ cmake path/to/aom 322 $ make 323 # It's best to build the testdata target using many make jobs. 324 # Running it like this will verify and download (if necessary) 325 # one at a time, which takes a while. 326 $ make testdata 327 $ path/to/aom/test/examples.sh --bin-path examples 328~~~ 329 330#### 3. Encoder tests: 331 332When making a change to the encoder run encoder tests to confirm that your 333change has a positive or negligible impact on encode quality. When running these 334tests the build configuration should be changed to enable internal encoder 335statistics: 336 337~~~ 338 $ cmake path/to/aom -DCONFIG_INTERNAL_STATS=1 339 $ make 340~~~ 341 342The repository contains scripts intended to make running these tests as simple 343as possible. The following example demonstrates creating a set of baseline clips 344for comparison to results produced after making your change to libaom: 345 346~~~ 347 # This will encode all Y4M files in the current directory using the 348 # settings specified to create the encoder baseline statistical data: 349 $ cd path/to/test/inputs 350 # This command line assumes that run_encodes.sh, its helper script 351 # best_encode.sh, and the aomenc you intend to test are all within a 352 # directory in your PATH. 353 $ run_encodes.sh 200 500 50 baseline 354~~~ 355 356After making your change and creating the baseline clips, you'll need to run 357encodes that include your change(s) to confirm that things are working as 358intended: 359 360~~~ 361 # This will encode all Y4M files in the current directory using the 362 # settings specified to create the statistical data for your change: 363 $ cd path/to/test/inputs 364 # This command line assumes that run_encodes.sh, its helper script 365 # best_encode.sh, and the aomenc you intend to test are all within a 366 # directory in your PATH. 367 $ run_encodes.sh 200 500 50 mytweak 368~~~ 369 370After creating both data sets you can use `test/visual_metrics.py` to generate a 371report that can be viewed in a web browser: 372 373~~~ 374 $ visual_metrics.py metrics_template.html "*stt" baseline mytweak \ 375 > mytweak.html 376~~~ 377 378You can view the report by opening mytweak.html in a web browser. 379 380 381### IDE hosted tests 382 383By default the generated projects files created by CMake will not include the 384runtests and testdata rules when generating for IDEs like Microsoft Visual 385Studio and Xcode. This is done to avoid intolerably long build cycles in the 386IDEs-- IDE behavior is to build all targets when selecting the build project 387options in MSVS and Xcode. To enable the test rules in IDEs the 388`ENABLE_IDE_TEST_HOSTING` variable must be enabled at CMake generation time: 389 390~~~ 391 # This example uses Xcode. To get a list of the generators 392 # available, run cmake with the -G argument missing its 393 # value. 394 $ cmake path/to/aom -DENABLE_IDE_TEST_HOSTING=1 -G Xcode 395~~~ 396 397### Downloading the test data 398 399The fastest and easiest way to obtain the test data is to use CMake to generate 400a build using the Unix Makefiles generator, and then to build only the testdata 401rule: 402 403~~~ 404 $ cmake path/to/aom -G "Unix Makefiles" 405 # 28 is used because there are 28 test files as of this writing. 406 $ make -j28 testdata 407~~~ 408 409The above make command will only download and verify the test data. 410 411### Adding a new test data file 412 413First, add the new test data file to the `aom-test-data` bucket of the 414`aomedia-testing` project on Google Cloud Platform. You may need to ask someone 415with the necessary access permissions to do this for you. 416 417NOTE: When a new test data file is added to the `aom-test-data` bucket, its 418"Public access" is initially "Not public". We need to change its 419"Public access" to "Public" by using the following 420[`gsutil`](https://cloud.google.com/storage/docs/gsutil_install) command: 421~~~ 422 $ gsutil acl ch -g all:R gs://aom-test-data/test-data-file-name 423~~~ 424This command grants the `AllUsers` group READ access to the file named 425"test-data-file-name" in the `aom-test-data` bucket. 426 427Once the new test data file has been added to `aom-test-data`, create a CL to 428add the name of the new test data file to `test/test_data_util.cmake` and add 429the SHA1 checksum of the new test data file to `test/test-data.sha1`. (The SHA1 430checksum of a file can be calculated by running the `sha1sum` command on the 431file.) 432 433### Additional test data 434 435The test data mentioned above is strictly intended for unit testing. 436 437Additional input data for testing the encoder can be obtained from: 438https://media.xiph.org/video/derf/ 439 440### Sharded testing 441 442The AV1 codec library unit tests are built upon gtest which supports sharding of 443test jobs. Sharded test runs can be achieved in a couple of ways. 444 445#### 1. Running test\_libaom directly: 446 447~~~ 448 # Set the environment variable GTEST_TOTAL_SHARDS to control the number of 449 # shards. 450 $ export GTEST_TOTAL_SHARDS=10 451 # (GTEST shard indexing is 0 based). 452 $ seq 0 $(( $GTEST_TOTAL_SHARDS - 1 )) \ 453 | xargs -n 1 -P 0 -I{} env GTEST_SHARD_INDEX={} ./test_libaom 454~~~ 455 456To create a test shard for each CPU core available on the current system set 457`GTEST_TOTAL_SHARDS` to the number of CPU cores on your system minus one. 458 459#### 2. Running the tests via the CMake build: 460 461~~~ 462 # For IDE based builds, ENABLE_IDE_TEST_HOSTING must be enabled. See 463 # the IDE hosted tests section above for more information. If the IDE 464 # supports building targets concurrently tests will be sharded by default. 465 466 # For make and ninja builds the -j parameter controls the number of shards 467 # at test run time. This example will run the tests using 10 shards via 468 # make. 469 $ make -j10 runtests 470~~~ 471 472The maximum number of test targets that can run concurrently is determined by 473the number of CPUs on the system where the build is configured as detected by 474CMake. A system with 24 cores can run 24 test shards using a value of 24 with 475the `-j` parameter. When CMake is unable to detect the number of cores 10 shards 476is the default maximum value. 477 478## Coding style 479 480We are using the Google C Coding Style defined by the 481[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). 482 483The coding style used by this project is enforced with clang-format using the 484configuration contained in the 485[.clang-format](https://chromium.googlesource.com/webm/aom/+/master/.clang-format) 486file in the root of the repository. 487 488You can download clang-format using your system's package manager, or directly 489from [llvm.org](http://llvm.org/releases/download.html). You can also view the 490[documentation](https://clang.llvm.org/docs/ClangFormat.html) on llvm.org. 491Output from clang-format varies by clang-format version, for best results your 492version should match the one used on Jenkins. You can find the clang-format 493version by reading the comment in the `.clang-format` file linked above. 494 495Before pushing changes for review you can format your code with: 496 497~~~ 498 # Apply clang-format to modified .c, .h and .cc files 499 $ clang-format -i --style=file \ 500 $(git diff --name-only --diff-filter=ACMR '*.[hc]' '*.cc') 501~~~ 502 503Check the .clang-format file for the version used to generate it if there is any 504difference between your local formatting and the review system. 505 506Some Git installations have clang-format integration. Here are some examples: 507 508~~~ 509 # Apply clang-format to all staged changes: 510 $ git clang-format 511 512 # Clang format all staged and unstaged changes: 513 $ git clang-format -f 514 515 # Clang format all staged and unstaged changes interactively: 516 $ git clang-format -f -p 517~~~ 518 519## Submitting patches 520 521We manage the submission of patches using the 522[Gerrit](https://www.gerritcodereview.com/) code review tool. This tool 523implements a workflow on top of the Git version control system to ensure that 524all changes get peer reviewed and tested prior to their distribution. 525 526### Login cookie 527 528Browse to [AOMedia Git index](https://aomedia.googlesource.com/) and login with 529your account (Gmail credentials, for example). Next, follow the 530`Generate Password` Password link at the top of the page. You’ll be given 531instructions for creating a cookie to use with our Git repos. 532 533### Contributor agreement 534 535You will be required to execute a 536[contributor agreement](http://aomedia.org/license) to ensure that the AOMedia 537Project has the right to distribute your changes. 538 539### Testing your code 540 541The testing basics are covered in the [testing section](#testing-the-av1-codec) 542above. 543 544In addition to the local tests, many more (e.g. asan, tsan, valgrind) will run 545through Jenkins instances upon upload to gerrit. 546 547### Commit message hook 548 549Gerrit requires that each submission include a unique Change-Id. You can assign 550one manually using git commit --amend, but it’s easier to automate it with the 551commit-msg hook provided by Gerrit. 552 553Copy commit-msg to the `.git/hooks` directory of your local repo. Here's an 554example: 555 556~~~ 557 $ curl -Lo aom/.git/hooks/commit-msg https://chromium-review.googlesource.com/tools/hooks/commit-msg 558 559 # Next, ensure that the downloaded commit-msg script is executable: 560 $ chmod u+x aom/.git/hooks/commit-msg 561~~~ 562 563See the Gerrit 564[documentation](https://gerrit-review.googlesource.com/Documentation/user-changeid.html) 565for more information. 566 567### Upload your change 568 569The command line to upload your patch looks like this: 570 571~~~ 572 $ git push https://aomedia-review.googlesource.com/aom HEAD:refs/for/master 573~~~ 574 575### Incorporating reviewer comments 576 577If you previously uploaded a change to Gerrit and the Approver has asked for 578changes, follow these steps: 579 5801. Edit the files to make the changes the reviewer has requested. 5812. Recommit your edits using the --amend flag, for example: 582 583~~~ 584 $ git commit -a --amend 585~~~ 586 5873. Use the same git push command as above to upload to Gerrit again for another 588 review cycle. 589 590In general, you should not rebase your changes when doing updates in response to 591review. Doing so can make it harder to follow the evolution of your change in 592the diff view. 593 594### Submitting your change 595 596Once your change has been Approved and Verified, you can “submit” it through the 597Gerrit UI. This will usually automatically rebase your change onto the branch 598specified. 599 600Sometimes this can’t be done automatically. If you run into this problem, you 601must rebase your changes manually: 602 603~~~ 604 $ git fetch 605 $ git rebase origin/branchname 606~~~ 607 608If there are any conflicts, resolve them as you normally would with Git. When 609you’re done, reupload your change. 610 611### Viewing the status of uploaded changes 612 613To check the status of a change that you uploaded, open 614[Gerrit](https://aomedia-review.googlesource.com/), sign in, and click My > 615Changes. 616 617## Support 618 619This library is an open source project supported by its community. Please 620please email aomediacodec@jointdevelopment.kavi.com for help. 621 622## Bug reports 623 624Bug reports can be filed in the Alliance for Open Media 625[issue tracker](https://bugs.chromium.org/p/aomedia/issues/list). 626