Conversation
Codecov Report
@@ Coverage Diff @@
## main #555 +/- ##
==========================================
+ Coverage 32.40% 37.29% +4.88%
==========================================
Files 27 27
Lines 3357 3357
Branches 756 756
==========================================
+ Hits 1088 1252 +164
+ Misses 2173 1989 -184
- Partials 96 116 +20
Flags with carried forward coverage won't be shown. Click here to find out more. |
| AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }} | ||
| OAI_CONFIG_LIST: ${{ secrets.OAI_CONFIG_LIST }} | ||
| run: | | ||
| coverage run -a -m pytest test/agentchat/contrib/test_math_user_proxy_agent.py |
There was a problem hiding this comment.
Looks like the output of coverage run will overwrite the output of the former step. In the end, only one step of coverage will be uploaded.
There was a problem hiding this comment.
Could we add different names to the output of different steps and combine them at the upload step?
There was a problem hiding this comment.
The coverage run -a will append outputs to the same file(.coverage), so we are good with this. Not very sure but this is what I find.
|
|
||
| jobs: | ||
| RetrieveChatTest: | ||
| OpenAI4ContribTests: |
There was a problem hiding this comment.
One problem with this approach is that the previous installed dependencies will remain in the later steps. That makes the test environment not clean for later steps. Is it possible to reset the environment for later steps?
There was a problem hiding this comment.
I haven't found any good way to clear up the environment. A possible solution is to use virtual env, but different os needs different syntax, which makes it complex. The other way is just putting them in different jobs.
I guess there are two problems to consider here:
- Is it ok to NOT having a clean env? Will there be a case that user does
pip install autogen[retrievechat, mathchat], and they expect things just work? - I suggested that separating the jobs would be tedious to look in github actions, which is why I change this in the first place. But we might need to revisit the problem: do you think it is indeed a issue, or we can bear with it?
There was a problem hiding this comment.
I think it's pretty important for the automatic jobs to test different contributed agents in different environments, each with just its own dependencies, whether this is done through separate virtual envs, different jobs, or some other mechanism. Wish I had more experience in setting up such tests.
There was a problem hiding this comment.
coverage run -a -m pytest test/test_retrieve_utils.py test/agentchat/contrib
will test all files in the contrib folder, while only dependencies for Retrievechat is installed. This will result in error in the future if a contrib agent requires other packages to be installed.
https://github.com/microsoft/autogen/blob/fda7a39dd9ede1c115d37c2a454f55b7f22c8193/.github/workflows/contrib-tests.yml#L52C38-L52C38
Why are these changes needed?
Example that previous test fails doesn’t block sequential tests
https://github.com/microsoft/autogen/actions/runs/6759692725/job/18372611806
(Install packages and dependencies for RetrieveChat -> fails
Run “Install packages and dependencies for MathChat” -> success
Run “test MathChat” -> success
Result is still error
)
When previous dependency fails, corresponding testing will be disabled.
https://github.com/microsoft/autogen/actions/runs/6759701280/job/18372633270?pr=555
(Run “ Install packages and dependencies for RetrieveChat” -> fail
Run “”Install packages and dependencies for MathChat ->fail
skipping test MathChat)
Related issue number
Checks