firebase-ios-sdk/scripts/health_metrics

SDK Health Metrics Report Generation

This directory includes code coverage report generation and binary size report generation.

Code Coverage Report Generation

This tool is to help generate coverage reports for pull requests. It's defined by the test_coverage workflow.

Coverage reports of SDK frameworks will be displayed in a pull request if the change is under corresponding SDK file patterns.

UpdatedFilesCollector will detect file changes and compare file paths to file patterns in file_patterns.json. If updated file paths fit any patterns, corresponding SDK coverage job will be triggered.

Add a new coverage workflow

To create a code coverage workflow for a new SDK,

1. Add newsdk and its patterns in file_patterns.json.
2. Add a new output flag, e.g. newsdk_run_job, in the coverage workflow. newsdk_run_job should be aligned with the name of SDK newsdk in file_patterns.json.

3. Add a newsdk coverage job in the workflow:

   pod-lib-lint-newsdk:
   needs: check
   # Don't run on private repo unless it is a PR.
   if: always() && github.repository == 'Firebase/firebase-ios-sdk' && (needs.check.outputs.newsdk_run_job == 'true'|| github.event.pull_request.merged)
   runs-on: macOS-latest
   strategy:
     matrix:
       target: [iOS]
   steps:
   - uses: actions/checkout@v3
   - uses: ruby/setup-ruby@359bebbc29cbe6c87da6bc9ea3bc930432750108
     with:
       ruby-version: '2.7'
   - name: Setup Bundler
     run: scripts/setup_bundler.sh
   - name: Build and test
     run: ./scripts/health_metrics/pod_test_code_coverage_report.sh FirebaseNewSDK "${{ matrix.target }}"
   - uses: actions/upload-artifact@v2
     with:
       name: codecoverage
       path: /Users/runner/*.xcresult
   

   Add the job name to the needs of create_report job.

4. If this newsdk podspec has unit test setup, e.g. database, with unit_tests.scheme = { :code_coverage => true }, the code coverage workflow should run unit tests through podspecs and utilize those coverage data and nothing is needed to update here. Otherwise, add another way of running tests and generating xcresult bundles with code coverage data in pod_test_code_coverage_report.sh.

Binary Size Report Generation

This tool is to generate a report for SDK binary sizes. create_binary_size_report.sh will trigger BinarySizeReportGenerator which will run cocoapods-size to create a binary size report and either post it in PRs in presubmits or upload it to the Metrics Service.

The cocoapods-size tool measures an iOS SDK size by comparing the 64 bit ARM slice before and after installing the pod to an iOS skeleton app. More about CocoaPods Size measurement could be found here.

Replicate a binary size measurement for a released SDK

To replicate an SDK binary size measurement locally, we can run the following script to get the binary size of the latest released FirebaseABTesting SDK.

git clone https://github.com/google/cocoapods-size
cd cocoapods-size
python3 measure_cocoapod_size.py --cocoapods FirebaseABTesting

Environment Setup

The binary_size_metrics on GitHub Actions workflow is running binary size measurement presubmits and postsubmits. The environment for this job could be found in runs-on. More details about the envionment could be found here.

Factors below might also affect the binary size:

• Each testapp is measured by a default version Xcode from the GitHub Actions environment above.
• Architecture is set to arm64 and the destination is generic/platform=iOS Simulator when a testapp is built.
• The source of dependencies is the SpecStaging repo, which is for prerelease, and CocoaPods CDN.