Generating Coverage Data

JavaScript / TypeScript with Jest

Jest has built-in coverage reporting capabilities using Istanbul under the hood.

$
# Add to your package.json or run directly
>
"scripts": {
>
  "test": "jest --coverage"
>
}

By default, Jest will output coverage reports to the coverage directory, including an LCOV report at coverage/lcov.info.

React

If you’re using Create React App, Jest is already configured. Simply run:

$
npm test -- --coverage

Node.js with NYC (Istanbul)

$
# Install NYC
>
npm install --save-dev nyc
>
>
# Add to your package.json
>
"scripts": {
>
  "test": "nyc mocha"
>
}

NYC will generate an LCOV report at coverage/lcov.info.

Python with pytest

$
# Install pytest-cov
>
pip install pytest-cov
>
>
# Run tests with coverage
>
pytest --cov=. --cov-report=xml

This will generate a coverage report in Cobertura XML format at coverage.xml.

Python with coverage.py

$
# Install coverage.py
>
pip install coverage
>
>
# Run your tests with coverage
>
coverage run -m pytest
>
>
# Generate an XML report
>
coverage xml

This will also generate a coverage report in Cobertura XML format at coverage.xml.

Ruby with SimpleCov

Add SimpleCov to your Gemfile:

1
group :test do
2
  gem 'simplecov', require: false
3
end

Configure SimpleCov at the top of your test helper file (e.g., test/test_helper.rb or spec/spec_helper.rb):

1
require 'simplecov'
2
SimpleCov.start
3
4
# Rest of your test helper file...

Run your tests normally:

$
bundle exec rspec
>
# or
>
bundle exec rails test

SimpleCov will generate a coverage report at coverage/.resultset.json.

Go

Go has built-in coverage capabilities:

$
# Run tests with coverage
>
go test -coverprofile=coverage.out ./...
>
>
# Convert to Clover format for Qlty using goclover
>
# First install goclover: go install github.com/t-yuki/goclover@latest
>
goclover coverage.out > clover.xml

This will generate a Clover XML report which Qlty can understand.

Java / Kotlin with Gradle and JaCoCo

Add JaCoCo to your Gradle build:

1
plugins {
2
    id 'jacoco'
3
}
4
5
jacoco {
6
    toolVersion = "0.8.8"
7
}
8
9
jacocoTestReport {
10
    reports {
11
        xml.required = true
12
        csv.required = false
13
        html.required = true
14
    }
15
}

Run tests and generate a JaCoCo report:

$
./gradlew test jacocoTestReport

This will generate an XML report at build/reports/jacoco/test/jacocoTestReport.xml.

Java / Kotlin with Maven and JaCoCo

Add JaCoCo to your Maven pom.xml:

1
<plugin>
2
    <groupId>org.jacoco</groupId>
3
    <artifactId>jacoco-maven-plugin</artifactId>
4
    <version>0.8.8</version>
5
    <executions>
6
        <execution>
7
            <goals>
8
                <goal>prepare-agent</goal>
9
            </goals>
10
        </execution>
11
        <execution>
12
            <id>report</id>
13
            <phase>test</phase>
14
            <goals>
15
                <goal>report</goal>
16
            </goals>
17
        </execution>
18
    </executions>
19
</plugin>

Run tests and generate a JaCoCo report:

$
mvn clean test

This will generate an XML report at target/site/jacoco/jacoco.xml.

PHP with PHPUnit

Configure PHPUnit in your phpunit.xml:

1
<phpunit>
2
    <!-- ... other configuration ... -->
3
    <coverage>
4
        <report>
5
            <clover outputFile="coverage.xml"/>
6
        </report>
7
    </coverage>
8
</phpunit>

Run PHPUnit:

$
phpunit --coverage-clover=coverage.xml

This will generate a Clover XML report at coverage.xml.

Swift Options

While there are multiple extraction options for Swift coverage data, each involves the same initial setup step: enable code coverage generation on your Xcode project or build. This can be done in your Xcode scheme, or via the xcodebuild command line option enableCodeCoverage YES.

By providing these settings, Xcode will generate coverage data into a binary format. Coverage data next needs to be exported into a format recognized by Qlty.

If you’re using Xcode 9.3 or later (recommended for most modern projects), we recommend using xccov to export coverage data, but you can also use Slather.

If using an older version of Xcode, Slather is likely your best option.

Using xccov (Xcode >= 9.3)

xccov was introduced in Xcode 9.3 as a simplified way to extract coverage data. xccov enables extracting coverage data from a single consolidated .xcresult bundle, greatly simplifying coverage setup. To use xccov:

1. Find / specify the location of the *.xcresult bundle

   If you are writing an Xcode build command yourself, you can supply it with the location where you’d like it to write the *.xcresult:

   $
   xcodebuild test -scheme MyTestProject -destination 'platform=macOS' -enableCodeCoverage YES -resultBundlePath ./TestResults.xcresult

   If instead you are utilizing a CI provider’s Xcode test step which then executes Xcode on your behalf, examine the underlying xcodebuild command it uses. It may provide a standardized location for the .xcresult bundle. If not, the step may export an environment variable containing the path to the .xcresult bundle.

2. Export coverage data using xccov

   Qlty supports the xccov format which is output when passed --archive and --json are passed as arguments as follows:

   $
   xcrun xccov view --archive --json TestResults.xcresult > xccov-coverage.json
   xccov supports two (poorly documented) coverage export JSON schemas generated with the command line arguments --archive and --report:

   The --archive format is the only supported format, as it’s the only format that contains information about which specific lines were executed and which were not.

   Below are examples of the two formats for reference:

   Archive format (supported)Report format (unsupported)

   1
   {
   2
       "/Users/ubuntu/code/qltysh/example-swift/Sources/MyTestProject/MyTestProject.swift": [
   3
           {
   4
           "isExecutable": false,
   5
           "line": 1
   6
           },
   7
           {
   8
           "executionCount": 5,
   9
           "isExecutable": true,
   10
           "line": 2
   11
           },
   12
           {
   13
           "isExecutable": false,
   14
           "line": 3
   15
           },
   16
           {
   17
           "executionCount": 3,
   18
           "isExecutable": true,
   19
           "line": 4
   20
           }
   21
       ]
   22
   }

3. When uploading this coverage data to Qlty, specify the format of the file as “xccov-json”

   For example, for the GitHub coverage action you would specify:

   1
       - name: Upload coverage report
   2
         uses: qltysh/qlty-action/coverage@ea1eaf434a27bf50cd544153084fbb11c52aaf84
   3
         with:
   4
           oidc: true
   5
           format: xccov-json
   6
           files: xccov-coverage.json

Swift with Xcode and Slather (see Slather’s documentation for supported Xcode versions)

Slather is a Ruby gem that extracts coverage data from Xcode projects.

On the pro side, Slather supports a wide range of Xcode versions, supports excluding files from coverage, and is fairly easy to use.

On the con side, Slather relies on lower level Apple tools like llvm-cov which is more complex and relies on being able to locate coverage-related files. On simple projects this may be fine. More complex multi-module projects need extra setup to ensure all coverage data is extracted. See Slather’s documentation for more information. In general the setup process is as follows:

1. Install Slather

   $
   gem install slather

2. Create a .slather.yml file:

   1
   coverage_service: cobertura_xml
   2
   xcodeproj: YourProject.xcodeproj
   3
   scheme: YourScheme
   4
   output_directory: coverage

3. Run your tests and then generate a coverage report:

   $
   xcodebuild -project YourProject.xcodeproj -scheme YourScheme -derivedDataPath Build/ -enableCodeCoverage YES test
   >
   slather

   This will generate a Cobertura XML report in the coverage directory.

4. When uploading coverage, specify the Cobertura format and provide the file generated by slather

Elixir with lcov_ex

Add lcov_ex to your dependencies in mix.exs:

1
defp deps do
2
  [
3
    {:lcov_ex, "~> 0.3", only: [:dev, :test], runtime: false}
4
  ]
5
end

Install the dependency:

$
mix deps.get

Option 1: Using the mix lcov task

Run the lcov task to run tests and generate coverage:

$
mix lcov

This will generate a coverage report at cover/lcov.info.

To output to a different folder:

$
mix lcov --output coverage

This will generate the report at coverage/lcov.info.

Option 2: Using mix test —cover

Alternatively, configure LcovEx as your test coverage tool in mix.exs:

1
def project do
2
  [
3
    app: :your_app,
4
    version: "0.1.0",
5
    elixir: "~> 1.15",
6
    deps: deps(),
7
    test_coverage: [tool: LcovEx, output: "coverage"]
8
  ]
9
end

Then run tests with coverage:

$
mix test --cover

This will generate a coverage report at coverage/lcov.info.

Umbrella Projects

For umbrella projects, running mix lcov at the umbrella level will generate coverage reports for all individual apps and compile them into a single file at cover/lcov.info.

To keep the individual app coverage files as well:

$
mix lcov --keep