Measuring the coverage of a rust program in Github Actions

After having faced a couple of of regressions in bo (my personal text editor written in Rust) in the past couple of days, I have tried to increase the number of unit tests related to the codebase sections handling navigation. I already had some unit tests, but I needed to know what lines of code were not tested, to know what area of the codebase I needed to focus on.

To do this, I used Mozilla's excellent grcov project. I followed their instructions and ran the following commands locally, in my work directory.

$ export RUSTFLAGS="-Cinstrument-coverage"
$ cargo build
$ export LLVM_PROFILE_FILE="bo-%p-%m.profraw"
$ cargo test
$ grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./target/debug/coverage/
$ open ./target/debug/coverage/index.html

This way, I got a beautiful HTML report in which I could see my code coverage, either global, file by file,

Coverage report

or line by line.

Coverage report

grcov even generates nice SVG badges displaying the coverage score, that I could display on the project homepage!

What I ultimately wanted though, was to have every commit touching my main branch to trigger a new coverage generation report, that I could host somewhere public and read at leisure when I needed to.

To do so, I set-up a publicly accessible s3 bucket, configured to host a static website, which turns out to be remarkably easy to do in terraform:

resource "aws_s3_bucket" "github-brouberol-coverage" {
  bucket        = "my-bucket-name"
  provider      = aws.euwest
  acl           = "public-read"
  force_destroy = false
  versioning {
    enabled    = false
    mfa_delete = false
  }
  website {
    index_document = "index.html"
  }
}

There are other ways to host the HTML files than S3 (such as Github Pages), and you do not have you terraform to do it, but I so happen to have a terraform codebase for my personal infrastructure, which made it a no-brainer. If you decide do host the files another way, feel free to jump ahead.

I then created an AWS user, associated with an AWS access_key/secret_key pair and the following IAM policy, granting that user read/write permissions on that S3 bucket, and nothing else.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::<my-bucket-name>",
                "arn:aws:s3:::<my-bucket-name>/*"
            ]
        }
    ]
}

I then had to store the bucket name, keypair and AWS region name as encrypted secrets in the bo repository, by going to Settings > Secrets > Actions > New repository secret.

Secrets

Once that was all set up, the project CI (Github Actions) needed to perform the following actions:

  • Checking out the project and setting up a nightly rust toolchain
- uses: actions/checkout@v2
- name: Setup toolchain
  uses: actions-rs/toolchain@v1
  with:
    toolchain: nightly
    override: true
    profile: minimal
  • running the unit tests with profiling and coverage collection enabled
- name: Run tests
  uses: actions-rs/cargo@v1
  with:
    command: test
    args: --all-features --no-fail-fast  # Customize args for your own needs
  env:
    CARGO_INCREMENTAL: '0'
    RUSTFLAGS: |
      -Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code
      -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests -Cinstrument-coverage
    RUSTDOCFLAGS: |
      -Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code
      -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests -Cinstrument-coverage'
  • generating the coverage report using grcov, using the actions-rs/grcov action.
- name: Gather coverage data
  id: coverage
  uses: actions-rs/grcov@v0.1
  • measuring the total coverage score, and report it in a check, if the job is associated to a pull request
- name: Report coverage in PR status for the current commit
  if: github.ref_name != 'main'
  run: |
    set -eu
    total=$(cat ${COV_REPORT_DIR}/badges/flat.svg | egrep '<title>coverage: ' | cut -d: -f 2 | cut -d% -f 1 | sed 's/ //g')
    curl -s "https://brouberol:${GITHUB_TOKEN}@api.github.com/repos/brouberol/bo/statuses/${COMMIT_SHA}" -d "{\"state\": \"success\",\"target_url\": \"https://github.com/brouberol/bo/pull/${PULL_NUMBER}/checks?check_run_id=${RUN_ID}\",\"description\": \"${total}%\",\"context\": \"Measured coverage\"}"
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    COMMIT_SHA: ${{ github.event.pull_request.head.sha }}
    RUN_ID: ${{ github.run_id }}
    PULL_NUMBER: ${{ github.event.pull_request.number }}
    COV_REPORT_DIR: ${{ steps.coverage.outputs.report }}

Secrets

  • uploading the whole HTML coverage report to S3, using the jakejarvis/s3-sync-action action. We only do this for commits belonging the main branch (i.e. direct pushes or after a pull request was merged).
- name: "Upload the HTML coverage report to S3"
  if: github.ref_name == 'main'
  uses: jakejarvis/s3-sync-action@master
  with:
    args: --acl public-read --follow-symlinks --delete
  env:
    AWS_S3_BUCKET: ${{ secrets.AWS_BUCKET }}
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    AWS_REGION: ${{ secrets.AWS_REGION }}
    SOURCE_DIR: ${{ steps.coverage.outputs.report }}
    DEST_DIR: 'bo'

With all of that set up, the coverage report is now publicly available, refreshed every time a new commit hits main, and I even get a coverage shield for free! coverage shield

Comments