This is a text-only version of the following page on https://raymii.org: --- Title : Github Actions, C++ with Boost and cmake, almost a 50% speedup with caching Author : Remy van Elst Date : 27-05-2020 URL : https://raymii.org/s/articles/Github_Actions_cpp_boost_cmake_speedup.html Format : Markdown/HTML --- For a personal project I use Github for source code hosting and Github Actions as an automated build and test tool. Github Actions compiles my `cmake` project and runs all the unit tests on every commit. It also saves a build artifact, the actual compiled program. By utilizing some dependency caching and make flags I sped up the build process by 43% by caching the `apt install libboost1.65-dev` and giving `cmake` a `-j2` makeflag. ![build speedup][1] > The improvements to the build script show the faster build time This article shows my simple setup to compile a C++ project with cmake and Boost on Github Actions. After compilation, it runs all the tests and uploads the compiled binary for download. For my one man project it's overkill, but when collaborating or when builds take a long time on your own machine, it's great to have an automated build / test system.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!

Do note that the build time decreased from 1 minute 48 seconds to 47 seconds for a small C++ project. The percentage wise speedup is large, but probably you might find the title a bit clickbaity. The main focus of this article is to show how to build a simple C++ project with Boost included using github actions. It also shows how to cache an `apt install` and how to provide `cmake` with the `MAKEFLAGS` to utilize the two cores that the free github builder virtual machine has. At work we use Gitlab CI for this and it cuts compilation time of the entire project from 2 hours to 20 minutes due to humongous build servers running gitlab runners. A few different binaries are built for different arm architectures, the test suite is run, doxygen docs are generated, code style checks are done and static analysis is done with Sonarqube, all from one source. With a team of developers this all gives an enormous speed increase in the process of reviewing code and not forgetting certain things. I don't have my own gitlab server running (anymore) but I noticed that github also have a feature like gitlab ci, but they call it Github Actions, and it's free for public projects, for private projects you get a limited amount of time, but 2000 minutes is enough for me. ### Simple cmake C++ project with Boost on Github Actions If you host your source code on github, you can use Github Actions. Most of my personal projects follow [this simple cmake structure][2] which integrates well with my preferred IDE, CLion by JetBrains. The structure also has unit tests with GoogleTest. For Boost integration, [check my other article][3] on integrating that in the project setup. On Ubuntu you also need to install the development libraries: apt install libboost-dev-all The Github linux virtual machine that will build the project does have most C++ development tools installed (like `gcc` and the `build-essential` package) but boost is missing. In the file you write which specifies your build steps you can also use `sudo` to install packages via `apt`, in our case `boost`. #### Basic workflow In the root folder of your project, create a folder for the workflow files for github: mkdir -p .github/workflows In that folder, create a `.yml` file for your workflow. My basic example to run `cmake` and my unit test is listed below. name: build and run tests on: push: branches: [ master ] pull_request: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 # install dependencies - name: boost run: sudo apt-get update && sudo apt-get install -yq libboost1.65-dev # build project - name: mkdir run: mkdir build - name: cmake build run: cmake -Bbuild -H. - name: cmake make run: cmake --build build/ --target all # run tests - name: run test 1 run: build/tst/Example1_tst - name: run test 2 run: build/tst/Example2_tst If you commit and push, you should be able to look up the action on Github: ![github action][4] That was easy wasn't is? A remote server builds your program and runs the unit tests. If you would do this on your local workstation the steps would be a bit like: #build code cd to/project/folder cd build cmake .. make # run tests tst/Example1_tst tst/Example2_tst #### Caching the apt install dependencies In my case the `apt update && apt install libboost-1.65-dev` takes almost 15 seconds. If you have more packages, this takes longer and its also run every time, but almost never changes. So a bit of a waste of time and resources. [This][5] post on Stackoverflow has an elaborate example on caching `apt` steps. My example is a simplified version. Replace this step in your workflow file: - name: boost run: sudo apt-get update && sudo apt-get install -yq libboost1.65-dev With the following piece of code: - name: Cache boost uses: actions/cache@v1.0.3 id: cache-boost with: path: "~/boost" key: libboost1.65-dev - name: Install boost env: CACHE_HIT: ${{steps.cache-boost.outputs.cache-hit}} run: | if [[ "$CACHE_HIT" == 'true' ]]; then sudo cp --force --recursive ~/boost/* / else sudo apt-get update && sudo apt-get install -yq libboost1.65-dev mkdir -p ~/boost for dep in libboost1.65-dev; do dpkg -L $dep | while IFS= read -r f; do if test -f $f; then echo $f; fi; done | xargs cp --parents --target-directory ~/boost/ done fi What this basically does is, if boost is not installed yet, install it and then use `dpkg` to copy all newly installed files to a folder. The next time, the virtual machine will download that `artifact` and just extract it on `/`. The effect is the same, the libraries are installed, however the time it takes is just 1 second instead of 15 seconds. If you need to install a newer version of the package, say, `libboost-1.71-dev`, replace the package name by the newer one and you're done. If you have multiple packages to install, make sure they're the actual packages, not a meta-package (a package without files, just dependencies). Meta-packages don't have files to copy, so the steps will fail. You can use the Ubuntu or Debian packages site to check, for example [libboost-dev][6] is a meta-package (10 kB package size, [no actual files][9]) where as [libboost1.71-dev][7] is an actual package. Larger file size and [lots of included files][8]. With this first improvement, subsequent build will be faster, especially when you have lots of dependencies to install. One more optimalisation we can do is provide a `makeflag` to use more resources during building. #### Provide makeflags to cmake In a cmake project, the build steps can all be done using cmake itself instead of the build system cmake generates for (like make/ninja), [if your cmake version is 3.15 or higher][10]): cd to/project/folder cmake --build build/ sudo cmake --install build/ No seperate `make`, the last cmake command wraps around that. You can also just do it the old fashioned way: cd to/project/folder/build cmake .. make all sudo make install Using the `cmake` commands works not only for `Makefiles`, but also for `ninja` or any other build system `cmake` can generate. But, in our example, we use `Makefiles` and to use [the two cores][11] the github virtual machine has (instead of just one core) we must provide a flag to `make`. If you would do it with the commandline you would do this: make -j2 all Where `-j#` is the amount of cores you want to use to build. Now with cmake we can do more complicated things in our `CMakeLists.txt`, but that would clutter up our simple example. Github Actions allows you to set environment variables and `make` can use the `MAKEFLAGS` environment variable. If we set that to contain `-j2`, even via `cmake`, the flag will be passed through. In our github actions yaml file, replace the following step: - name: cmake make run: cmake --build build/ --target all With the following code. You could also just add the last two lines instead of replacing the whole block. - name: cmake make run: cmake --build build/ --target all env: MAKEFLAGS: "-j2" In my case using two cores sped up the build process by another 27 seconds. If your project is larger, the improvement will be bigger as well. ### Upload build artifacts One of the other usefull features is to be able to download certain files that were built. Github calls them `build artifacts` and you can download them via the webpage: ![gh action download][12] At work, via Gitlab, we use this to cross compile for a few different ARM architectures. Not everybody has a crosscompiler setup, but they can just download their freshly built binary and run it on actual hardware. Most of our testing is automated with unit tests, but there are edge cases, for example, interaction with actual hardware (think valves, pumps, high voltage relais). If you don't crosscompile, it is still useful, it allows other people to get a binary without having to compile it. A tester could login, download the binary for their specific feature branch and use it for testing. Build artifacts are also reproducable. You can trigger a build of a branch from 6 months ago and get that binary, just as pristine as it was back then. Add the following to the bottom of your yml file. The paths are for our example. # upload artifact, example binary - name: Upload Example binary uses: actions/upload-artifact@v1 with: name: upload binary path: build/src/Example You can go crazy with this, couple it with github releases for certain branches and automate more, but that is out of scope for our example case. #### The final yaml file The yaml file with all improvements is listed below: name: build and run tests on: push: branches: [ master ] pull_request: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 # install and cache dependencies - name: Cache boost uses: actions/cache@v1.0.3 id: cache-boost with: path: "~/boost" key: libboost1.65-dev - name: Install boost env: CACHE_HIT: ${{steps.cache-boost.outputs.cache-hit}} run: | if [[ "$CACHE_HIT" == 'true' ]]; then sudo cp --force --recursive ~/boost/* / else sudo apt-get update && sudo apt-get install -yq libboost1.65-dev mkdir -p ~/boost for dep in libboost1.65-dev; do dpkg -L $dep | while IFS= read -r f; do if test -f $f; then echo $f; fi; done | xargs cp --parents --target-directory ~/boost/ done fi # build project - name: mkdir run: mkdir build - name: cmake build run: cmake -Bbuild -H. - name: cmake make run: cmake --build build/ --target all env: MAKEFLAGS: "-j2" # run tests - name: run test 1 run: build/tst/Example1_tst - name: run test 2 run: build/tst/Example2_tst # upload artifact, game binary - name: Upload Example binary uses: actions/upload-artifact@v1 with: name: upload binary path: build/src/Example ### Conclusion This article discussed both the automated build setup of a `C++` project on Github actions, how to upload build artifacts and two improvements to speed up such a build. In my case the improvements are significant percentage wise, but not that impressive if you look at the actual numbers. In the case of larger projects, or when you are billed for runtime, the improvements could have a bigger effect. [1]: /s/inc/img/gh-actions-3.png [2]: /s/tutorials/Cpp_project_setup_with_cmake_and_unit_tests.html [3]: /s/snippets/std_string_to_lowercase_or_uppercase_in_cpp.html#toc_4 [4]: /s/inc/img/gh-actions-2.png [5]: http://web.archive.org/web/20200526180814/https://stackoverflow.com/questions/59269850/caching-apt-packages-in-github-actions-workflow [6]: http://web.archive.org/web/20200526180847/https://packages.ubuntu.com/focal/libboost-dev [7]: http://web.archive.org/web/20200526180850/https://packages.ubuntu.com/focal/libboost1.71-dev [8]: http://web.archive.org/web/20200526180852/https://packages.ubuntu.com/focal/amd64/libboost1.71-dev/filelist [9]: http://web.archive.org/web/20200526180848/https://packages.ubuntu.com/focal/amd64/libboost-dev/filelist [10]: http://web.archive.org/web/20200526174351/https://cliutils.gitlab.io/modern-cmake/chapters/intro/running.html [11]: http://web.archive.org/web/20200526174717/https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources [12]: /s/inc/img/gh-actions-1.png --- License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source. This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it. All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Just to be clear, the information on this website is for meant for educational purposes and you use it at your own risk. I do not take responsibility if you screw something up. Use common sense, do not 'rm -rf /' as root for example. If you have any questions then do not hesitate to contact me. See https://raymii.org/s/static/About.html for details.