CircleCI 2.0 Parallel builds SimpleCov coverage report merging locally

CircleCI 2.0 Parallel builds SimpleCov coverage report merging locally overview.md

Simplecov aggregated coverage report from CircleCI 2.0 parallel builds (focused on storing locally/within CI containers as artifacts)

Problem Statement

We have Rails application which is running tests on circleCI 2.0, we have simplecov configured to track the coverage of our test suite. Now the problem is with parallelism enabled, we have partial coverage reports in all different containers according to the tests those containers ran.

We obviously want to have consolidated simplecov coverage report which actually shows us overall coverage report.

Solution(s)

There are a few steps involved in this process, we have to fetch all coverage jsons generated by simplecov in different containers of the build, and than merge those.

For fetching the coverage jsons we have two options:

1- Use CircleCI 2.0 feature of workflows. (This is more straight forward and makes more sense when you already are using workflows)

2- Use deploy step in simple CircleCI 2.0 build. Build step also waits for all parallel builds to finish before it executes this step in container number 0. (This method requires to use circleci api to fetch coverage files from builds)

**Credits: ** Help was taken from many resources main resources were this & this. Help from CircleCI support was also taken, although that was not very helpful but helped clear some things.

simplecov_merger.rb

#!/usr/bin/env ruby

# NOTE: This will only work if you have simplecov >= 0.16.1

require 'open-uri'
require 'active_support/inflector'
require 'simplecov'

class SimpleCovHelper
  def self.report_coverage(base_dir: './coverage_results')
    SimpleCov.start 'rails' do
      skip_check_coverage = ENV.fetch('SKIP_COVERAGE_CHECK', 'false')
      add_filter '/spec/'
      add_filter '/config/'
      add_filter '/vendor/'
      Dir['app/*'].each do |dir|
        add_group File.basename(dir).humanize, dir
      end
      minimum_coverage(100) unless skip_check_coverage
      merge_timeout(3600)
    end
    new(base_dir: base_dir).merge_results
  end

  attr_reader :base_dir

  def initialize(base_dir:)
    @base_dir = base_dir
  end

  def all_results
    Dir["#{base_dir}/.resultset*.json"]
  end

  def merge_results
    results = all_results.map { |file| SimpleCov::Result.from_hash(JSON.parse(File.read(file))) }
    SimpleCov::ResultMerger.merge_results(*results).tap do |result|
      SimpleCov::ResultMerger.store_result(result)
    end
  end
end

api_url = "https://circleci.com/api/v1.1/project/github/#{ENV['CIRCLE_PROJECT_USERNAME']}/#{ENV['CIRCLE_PROJECT_REPONAME']}/#{ENV['CIRCLE_BUILD_NUM']}/artifacts?circle-token=#{ENV['API_TOKEN']}"
artifacts = open(api_url)
coverage_dir = 'coverage_results'
SimpleCov.coverage_dir(coverage_dir)
JSON.load(artifacts)
    .map do |artifact|
      path = artifact['path']
      next if !(path.end_with?('/.resultset.json') && path.include?('/coverage/'))
      JSON.load(open("#{artifact['url']}?circle-token=#{ENV['API_TOKEN']}"))
    end.compact
    .each_with_index do |resultset, i|
      resultset.each do |_, data|
        result = SimpleCov::Result.from_hash(['command', i].join => data)
        SimpleCov::ResultMerger.store_result(result)
      end
    end

merged_result = SimpleCov::ResultMerger.merged_result
merged_result.command_name = 'RSpec'

SimpleCovHelper.report_coverage

Solution using deploy step.md

Using CircleCI 2.0 deploy step and circleci api v1.1 to merge SimpleCov coverage files of parallel builds

CircleCI 2.0 deploy step doc

Notes:

1. Using this we don't HAVE to use workflows.
2. We get to keep rebuild without cache button. (YAY!!)
3. We don't have to spin up another container, this work is done in container 0 which is already setup with all configurations etc. So it is bit faster.

Procedure:

I was able to configure this properly and get html report generated using parts from both gists i have mentioned in the main doc in this gist. Some issues i faced were that coverage was not realistically accurate, issue was that i was initializing simplecov a bit late, i move it above all other includes etc. This is literally how beginning looks in my spec_helper.rb file.

require 'simplecov'

if ENV['CIRCLE_ARTIFACTS']
  dir = File.join(ENV['CIRCLE_ARTIFACTS'], 'coverage')
  SimpleCov.coverage_dir(dir)
end
SimpleCov.start 'rails' do
  add_filter '/spec/'
  add_filter '/vendor/'
end

Merge code is in simplecov_merger.rb file(attached in this gist). I choose to run this ruby code directly instead of rake task. You can obviously choose however you want to use this code.

CirclieCI api token setting

You also have to generate a circleci API token for an account which has access to this project on circleci, and set that api token in an env_var in project settings. That environment variable will be available for you than with the name you choose. Replace API_TOKEN in the simplecov_merger.rb file with that environment variable. You probably don't want to commit this token to git as anyone having this token can do whatever he want on circleci.

CircleCI doc on environemtn variables and CircleCI doc for managing your API tokens

Note: To use this file as is without rake tasks, you have to give this file execute permission(using chmod) before committing it in your git repo.

Following is how my .circleci/config.yml file calls this code:

    # Store generated coverage for each build(lookout if you have set a different directory, in which case you will also have to change in rb file)
    - store_artifacts:
        path: coverage
        
    # Make directory to store aggregated/combined results
    - run:
        name: Stash Coverage Results
        command: |
          mkdir coverage_results
          
    # This is magic step, which actually will combine coverage reports of all parallel containers
    - deploy:
        name: Merge and copy coverage data
        command: |-
          RUN_COVERAGE=true bundle exec spec/simplecov_merger.rb
    # without RUN_COVERAGE=true you wont get html coverage report generated

    # Now store the results so we can see them in artifacts(coverage report will only be in container 0 as described earlier)
    - store_artifacts:
        path: /home/circleci/projDirec/projName/coverage_results

Solution using workflows.md

Using CircleCI 2.0 workflows to merge SimpleCov's generated coverage reports in parallel builds

CircleCI 2.0 Workflows Docs

Notes:

1. Intro: Workflows is a nice feature it defines a flow in which different steps of flow are performed. For example for Rspec tests there can be one step, on second step we can check some security problems say from brakeman or any other tool and in next step we can combine coverage reports(for which this gist is). And in another step we can deploy our app.
2. Every step in workflows initializes a new container and runs only when all previous steps have succeeded. You can define different parallelism and different configuration for each step.
3. You should also know that at the time of writing, there is no option of rerun without cache option when using workflows, circleci blogs say it is a feature request they are considering to add. But there is no timeline provided.

Procedure:

You can follow this issue for this approach, this gist from trev is provided in the above issue and this was what actually lead me to implement it.

I should say that if you are new to workflows, understanding the configurations and working of workflows can take some time. But at the end you will be able to get it working with these resources.

Only issues i encountered were because of cache keys etc so lookout for that and you will be fine.

Also i was not properly saving generated report and was assuming it is not generating report for some reason. Use ls and other commands to debug within your builds if you encounter such issues.

Firstly, Thanks for a detailed post. I followed all the steps that you have mentioned, but the problem was at the end it said '0 lines covered out of ... lines' . I guess this was because in the simplecov_merger.rb file you have a SimpleCov.start which was triggering a new start from this file and hence the coverage was being considered as 0. When I tried removing SimpleCov.start from this file (I was still having SimpleCov.start in spec_helper file), the coverage report was not generated at all. So I decided to change the approach and upgraded SimpleCov gem to 0.18.5 and used SimpleCov.collate method after generating files out of the JSON response from CIRCLE CI.

Modified simplecov_merger.rb

`
api_url = "[https://circleci.com/api/v1.1/project/](https://circleci.com/api/v1.1/project/%5C)///#{ENV['JOB_NUMBER']}/artifacts?circle-token=#{ENV['API_TOKEN']}"
puts 'API CALL TO FETCH ARTIFACTS FROM ALL BOXES :' + api_url
artifacts = open(api_url)
coverage_dir = 'coverage_results'
SimpleCov.coverage_dir(coverage_dir)
SimpleCov.merge_timeout(3600)
SimpleCov.command_name 'RSpec'
allArtifacts=JSON.load(artifacts)
puts 'ARTIFACTS RESPONSE :'
puts allArtifacts
allArtifacts
.map do |artifact|
path = artifact['path']
if (path.end_with?('/.resultset.json') && path.include?('coverage/'))
puts "File Name: #{path}"
node_index=artifact['node_index']
puts "node_index : #{node_index}"
json_payload=JSON.load(open("#{artifact['url']}?circle-token=#{ENV['API_TOKEN']}"))
Dir.mkdir("coverage-#{node_index}") unless Dir.exist?("coverage-#{node_index}")
File.write("coverage-#{node_index}/.resultset.json",JSON.dump(json_payload))
end
end

SimpleCov.collate Dir["coverage-*/.resultset.json"], 'rails' do
formatter SimpleCov::Formatter::MultiFormatter.new([
SimpleCov::Formatter::SimpleFormatter,
SimpleCov::Formatter::HTMLFormatter
])
end
`
This solved the issue and I was able to get the merged result in coverage_results folder of the 0th box.

Thankyou for sharing your solution, this post is a bit outdated, but with you comment it should be re-useable again. Thanks.

We ran into issues with our merging script on CircleCI as well after upgrading to simplecov 0.19.0. We got empty coverage reports as the merge result (0% coverage).

We were able to fix it. Have a look here: https://gist.github.com/trev/9cc964a54c8d5b62f4def891eba6b976#gistcomment-3436285