Use codecov-cli over python-codecov by FileMagic · Pull Request #6036 · StackStorm/st2

FileMagic · 2023-10-16T00:07:07Z

This is the PR related to Issue #6035

arm4b

Thanks for the PR to fix it!

Looks good,
I left one comment about verifying the coverage in this PR.
Please also update the Changelog with a note about using codecov-cli.

scripts/ci/submit-codecov-coverage.sh

FileMagic · 2023-10-17T23:25:37Z

A couple of things to note in the PR. The python 3.6 tests seem to be failing due to the codecov-cli-0.1.13 being installed which was intresting, so I am going to try upgrading the version of pip before to see if that helps.

The python 3.9 and python 3.8 versions seem to have some issue with getting the environment var CODECOV_TOKEN which is talked about here as the following:

Note that these commands will automatically search your environment for a $CODECOV_TOKEN environment variable and use it if found. If you do not have a repository upload token, or global upload token, stored as an environment variable, you will need to pass it into each command manually, like so: -t {$CODECOV_TOKEN}.

I don't see where this is being set in the code base or via the github action so I guess I'll try the --git-service which is not documented for the README.md, but seems to be in their help menu (could be a side effect of this being a combined cmd).

FileMagic · 2023-10-17T23:52:17Z

A couple of things to note in the PR. The python 3.6 tests seem to be failing due to the codecov-cli-0.1.13 being installed which was intresting, so I am going to try upgrading the version of pip before to see if that helps.

The python 3.9 and python 3.8 versions seem to have some issue with getting the environment var CODECOV_TOKEN which is talked about here as the following:

Note that these commands will automatically search your environment for a $CODECOV_TOKEN environment variable and use it if found. If you do not have a repository upload token, or global upload token, stored as an environment variable, you will need to pass it into each command manually, like so: -t {$CODECOV_TOKEN}.

I don't see where this is being set in the code base or via the github action so I guess I'll try the --git-service which is not documented for the README.md, but seems to be in their help menu (could be a side effect of this being a combined cmd).

Yeah after looking a bit further I now see where the env vars can be seen in github, and it's def not being passed from that end. If anyone know please just leave a comment, however, I do not know where to pull the codecov token from.

On the topic of python 3.6. It seems that the tool does not have the upload-process under python 3.6. So I guess we will see. Hopefully there won't be support for python 3.6 in st2 for much longer anyways, but this could be done using the fully written out commands nontheless...

FileMagic · 2023-10-18T18:21:03Z

Thanks for the commit, for my own docs in the future I didn't see this: (GitHub) Using secrets in a workflow.

I'll probs try using the -t flag later on today to see if I can get this working.

arm4b · 2023-10-18T18:31:40Z

@FileMagic We indeed didn't have CODECOV_TOKEN added (generated) for this repository.

I just added it, but looks like the main repository secrets are not accessible from the GH forks (codecov/codecov-action#44 (comment)) during the build.

Could you try to generate a Codecov token and add it to your Github Repo (fork) just for testing purposes?

arm4b · 2023-10-18T18:47:59Z

Minor addition, based on https://github.com/StackStorm/st2/actions/runs/6564663313/job/17831515543?pr=6036#step:17:33

warning - 2023-10-18 18:03:47,823 -- Config file codecov.yml not found, or is not a file. Ignoring config.

We may need to rename the .codecov.yml to codecov.yml as the file name format (I guess) was changed with the new Codecov.

FileMagic · 2023-10-20T00:13:16Z

It seems that even when CODECOV_TOKEN is set within the fork it does not work.

FileMagic · 2023-10-20T04:25:40Z

I used a key for code cov here one time on a free account to test if it would work. It seems that it doesn't read from my project's github secrets either. I regenerated the key after so we should be good. However, we will see if this technically would have worked.

FileMagic · 2023-10-20T13:18:05Z

I think the reason for why we are seeing:

info - 2023-10-20 04:33:00,063 -- ci service found: github-actions
info - 2023-10-20 04:33:00,247 -- Process Commit creating complete
error - 2023-10-20 04:33:00,248 -- Commit creating failed: {"detail":"You do not have permission to perform this action."}

Is due to the permissions being set to run on my repo, but I think that this is currently. Not quite sure though to be honest.

…pload

FileMagic · 2023-10-21T18:55:58Z

Well, that seems to work as expected. I'll revert the change ENABLE_COVERAGE here in a bit.

FileMagic · 2023-10-21T19:06:43Z

test-requirements.txt

I unpin the version of coverage here due to codecovcli not requiring it anymore directly. This can be seen here

FileMagic · 2023-10-21T19:30:36Z

The last thing I did was remove the extra coverage install from scripts/ci/submit-codecov-coverage.sh.

I tested the updated script (without coverage install) in my cli:

Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (23.0)
Collecting pip
  Downloading pip-23.3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 7.0 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0
    Uninstalling pip-23.0:
      Successfully uninstalled pip-23.0
Successfully installed pip-23.3.1
info - 2023-10-21 19:26:39,462 -- ci service found: local
info - 2023-10-21 19:26:39,611 -- Process Commit creating complete
error - 2023-10-21 19:26:39,611 -- Commit creating failed: {"detail":"Invalid token."}

So we should be good now. Let me know if you see anything else that should be changed.

arm4b · 2023-10-22T12:51:48Z

scripts/ci/submit-codecov-coverage.sh

-    # Without eventlet being available to the coverage command it will fail with "Couldn't trace with concurrency=eventlet, the module isn't installed."
-    pip install eventlet
-    # NOTE: codecov only supports coverage==4.5.2
-    pip install 'coverage<5.0'


With removing coverage (9f27017) in a bash script we get the following error at a later stage:

https://github.com/StackStorm/st2/actions/runs/6603759812/job/17937403741?pr=6040#step:17:38

info - 2023-10-22 12:48:45,887 -- ci service found: github-actions info - 2023-10-22 12:48:45,991 -- Process Commit creating complete info - 2023-10-22 12:48:46,101 -- Process Report creating complete info - 2023-10-22 12:48:46,101 -- Finished creating report successfully --- {"response": "{\"external_id\":\"2aa2e37f-d821-4cad-9199-ecd04092be89\",\"created_at\":\"2023-10-22T12:35:18.103083Z\",\"commit_sha\":\"d1666b936f8e58a8beccc69ad8d4a1772f74b20b\",\"code\":null}"} warning - 2023-10-22 12:48:46,102 -- xcrun is not installed or can't be found. warning - 2023-10-22 12:48:46,429 -- No gcov data found. warning - 2023-10-22 12:48:46,429 -- coverage.py is not installed or can't be found. info - 2023-10-22 12:48:46,522 -- Found 0 coverage files to upload Error: No coverage reports found. Please make sure you're generating reports successfully. Error: Process completed with exit code 1.

so it's indeed needed, please keep it included

Will put it back one sec here.

I apologize about that. That should do it for ya.

No worries, and thanks for your work! 👍

…cli" This reverts commit 456b61b.

arm4b

Thanks a lot for fixing the st2 master build! 🟢

arm4b · 2023-10-23T17:20:58Z

Unit Tests are running fine now and reporting the coverage 🟢 (https://app.codecov.io/gh/StackStorm/st2/commits) after merging this PR.

The Orquesta Integration tests though failed 🔴 on:

st2/.github/workflows/orquesta-integration-tests.yaml

Lines 58 to 75 in d374b92

    
           - name: 'Integration Tests (Orquesta)' 
        
             task: 'ci-orquesta' 
        
             nosetests_node_total: 1 
        
             nosetests_node_index: 0 
        
             python-version: '3.6.13' 
        
             python-version-short: '3.6' 
        
           - name: 'Integration Tests (Orquesta)' 
        
             task: 'ci-orquesta' 
        
             nosetests_node_total: 1 
        
             nosetests_node_index: 0 
        
             python-version-short: '3.8' 
        
             python-version: '3.8.10' 
        
           - name: 'Integration Tests (Orquesta)' 
        
             task: 'ci-orquesta' 
        
             nosetests_node_total: 1 
        
             nosetests_node_index: 0 
        
             python-version-short: '3.9' 
        
             python-version: '3.9.14'

st2/.github/workflows/orquesta-integration-tests.yaml

Lines 227 to 231 in d374b92

    
                 - name: Codecov 
        
                   # NOTE: We only generate and submit coverage report for master and version branches and only when the build succeeds (default on GitHub Actions, this was not the case on Travis so we had to explicitly check success) 
        
                   if: "${{ success() && env.ENABLE_COVERAGE == 'yes' }}" 
        
                   run: | 
        
                     ./scripts/ci/submit-codecov-coverage.sh

where we missed to include the CODECOV_TOKEN and py3.8 condition in GH workflows.

@FileMagic If you'd like to help with that as well, considering you already figured out the solution in this PR.

FileMagic · 2023-10-23T17:24:30Z

Unit Tests are running fine now and reporting the coverage 🟢 (https://app.codecov.io/gh/StackStorm/st2/commits) after merging this PR.

The Orquesta Integration tests though failed 🔴 on:

st2/.github/workflows/orquesta-integration-tests.yaml

Lines 58 to 75 in d374b92

- name: 'Integration Tests (Orquesta)'

task: 'ci-orquesta'

nosetests_node_total: 1

nosetests_node_index: 0

python-version: '3.6.13'

python-version-short: '3.6'

- name: 'Integration Tests (Orquesta)'

task: 'ci-orquesta'

nosetests_node_total: 1

nosetests_node_index: 0

python-version-short: '3.8'

python-version: '3.8.10'

- name: 'Integration Tests (Orquesta)'

task: 'ci-orquesta'

nosetests_node_total: 1

nosetests_node_index: 0

python-version-short: '3.9'

python-version: '3.9.14'

st2/.github/workflows/orquesta-integration-tests.yaml

Lines 227 to 231 in d374b92

- name: Codecov

# NOTE: We only generate and submit coverage report for master and version branches and only when the build succeeds (default on GitHub Actions, this was not the case on Travis so we had to explicitly check success)

if: "${{ success() && env.ENABLE_COVERAGE == 'yes' }}"

run: |

./scripts/ci/submit-codecov-coverage.sh

where we missed to include the CODECOV_TOKEN and py3.8 condition in GH workflows.
* https://github.com/StackStorm/st2/actions/runs/6612741025/job/17970691174

* https://github.com/StackStorm/st2/actions/runs/6612741025/job/17970691413

* https://github.com/StackStorm/st2/actions/runs/6612741025/job/17970691618
@FileMagic If you'd like to help with that as well, considering you already figured out the solution in this PR.

I see, yeah I can fix that tonight.

Use codecov-cli over python-codecov

8e4f965

pull-request-size bot added the size/S PR that changes 10-29 lines. Very easy to review. label Oct 16, 2023

arm4b suggested changes Oct 16, 2023

View reviewed changes

scripts/ci/submit-codecov-coverage.sh Show resolved Hide resolved

arm4b modified the milestones: 3.8.1, 3.9.0 Oct 16, 2023

FileMagic added 3 commits October 17, 2023 15:58

Add codecov to changelogs

341a49e

Enable coverage for the meantime to test codecov

46ab07f

enable coverage via the var instead in setup env

e002b73

FileMagic added 2 commits October 17, 2023 19:26

Remove pinning of coverage

4669ff9

Add some updates to codecov submission

0cc2ec5

pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/S PR that changes 10-29 lines. Very easy to review. labels Oct 17, 2023

set git service for codecovcli

043d439

arm4b added 2 commits October 18, 2023 18:26

Merge branch 'master' into fix_st2_codecov

9056e01

Add Codecov token to GH workflows

0f81454

Rename codecov config file

f224a2d

arm4b mentioned this pull request Oct 19, 2023

TSC Meeting (10 Oct 2023) - Fixing builds, Upstream dependencies, stackstorm.com, Python difficulties, new OS support StackStorm/community#125

Closed

13 tasks

FileMagic added 2 commits October 19, 2023 14:38

Use codecov action intead of bash script

87dc75b

Add token to codecovcli call

2121afd

FileMagic added 2 commits October 19, 2023 20:18

Add quotes to codecov

559a3e5

confirm ci-cd working using key in commit (will be invalid after).

98c3636

arm4b modified the milestones: 3.9.0, 3.8.1 Oct 21, 2023

FileMagic added 2 commits October 21, 2023 17:50

Remove nix shell.

29cc295

Set coverage to work only on python 3.8

7b837e0

pull-request-size bot added size/S PR that changes 10-29 lines. Very easy to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Oct 21, 2023

Add enviorment varibles to pull in python version for code coverage u…

9fe0f12

…pload

pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/S PR that changes 10-29 lines. Very easy to review. labels Oct 21, 2023

Revert ENABLE_COVERAGE upload change

5687d33

FileMagic commented Oct 21, 2023

View reviewed changes

Remove coverage install for it is no longer a dep to codecov-cli

456b61b

pull-request-size bot added size/S PR that changes 10-29 lines. Very easy to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Oct 21, 2023

arm4b suggested changes Oct 22, 2023

View reviewed changes

Revert "Remove coverage install for it is no longer a dep to codecov-…

337e427

…cli" This reverts commit 456b61b.

pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/S PR that changes 10-29 lines. Very easy to review. labels Oct 22, 2023

arm4b approved these changes Oct 23, 2023

View reviewed changes

arm4b requested a review from a team October 23, 2023 11:22

arm4b linked an issue Oct 23, 2023 that may be closed by this pull request

Fix CI / Unit Tests Failing #6035

Closed

cognifloyd approved these changes Oct 23, 2023

View reviewed changes

cognifloyd merged commit d374b92 into StackStorm:master Oct 23, 2023

FileMagic mentioned this pull request Oct 24, 2023

Fix Orquesta Integration Test Codecov Upload #6046

Merged

arm4b mentioned this pull request Oct 24, 2023

Remove unused codecov for Orquesta integration tests #6048

Merged

arm4b mentioned this pull request Nov 10, 2023

TSC Meeting (14 Nov 2023) - v3.8.1 patch release progress, v3.9.0 release plans StackStorm/community#127

Closed

13 tasks

Uh oh!

Conversation

FileMagic commented Oct 16, 2023

Uh oh!

arm4b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FileMagic commented Oct 17, 2023

Uh oh!

FileMagic commented Oct 17, 2023

Uh oh!

FileMagic commented Oct 18, 2023

Uh oh!

arm4b commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arm4b commented Oct 18, 2023

Uh oh!

FileMagic commented Oct 20, 2023

Uh oh!

FileMagic commented Oct 20, 2023

Uh oh!

FileMagic commented Oct 20, 2023

Uh oh!

FileMagic commented Oct 21, 2023

Uh oh!

FileMagic Oct 21, 2023

Choose a reason for hiding this comment

Uh oh!

FileMagic commented Oct 21, 2023

Uh oh!

arm4b Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FileMagic Oct 22, 2023

Choose a reason for hiding this comment

Uh oh!

FileMagic Oct 22, 2023

Choose a reason for hiding this comment

Uh oh!

arm4b Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

arm4b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arm4b commented Oct 23, 2023

Uh oh!

FileMagic commented Oct 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arm4b commented Oct 18, 2023 •

edited

Loading

arm4b Oct 22, 2023 •

edited

Loading

arm4b left a comment •

edited

Loading