Skip to content

Feature python integration tests#1134

Merged
dluc merged 10 commits intomainfrom
feature-python-integration-tests
May 21, 2023
Merged

Feature python integration tests#1134
dluc merged 10 commits intomainfrom
feature-python-integration-tests

Conversation

@awharrison-28
Copy link
Contributor

Motivation and Context

Python integration tests were failing consistently for 2 reasons. First, all ubuntu system tests were failing due to missing shared object files related to cuda. Second, macOS + python 3.11 tests were failing due to hsnwlib unable to install on native hardware.

See #1066 for description of changes to integration test code to condense test code, make better use of pytest fixtures, and reduce the number of calls to external APIs (OpenAI, AOAI)

Description

  • Integration tests were failing on Linux due to a regression in python torch=2.0.1. Downgrading the version to 2.0.0 resolved these failures.
  • For MacOS + 3.11 specifically, setting the environment variable HNSWLIB_NO_NATIVE=1 ensures that the hnswlib wheel can be built and installed properly on M1 hardware.
  • Python integration tests reliably and consistently passing: https://github.com/microsoft/semantic-kernel/actions/runs/5035748596

Contribution Checklist

awharrison-28 and others added 9 commits May 19, 2023 16:19
### Motivation and Context
This PR slims down the number of integration tests running against AOAI
and OAI models - reducing time and resources spent running them.
Additional justification for slimming down these tests is to reduce the
chance of throttling from these endpoints (leads to unstable integration
tests).

The scenarios previously being covered are already handled using HF
models which are much less expensive to test against and do not run the
risk of throttling.


### Description
- added a top-level test `conftest.py` to handle kernel creation and
OpenAI model secret handling. I had originally intended `create_kernel`
to do more than just create a kernel, but additional setup wasn't
needed. I've left the pytest fixture though since other fixtures can
call it, and using it in tests can making importing Kernel from
semantic_kernel unncessary.
- added a completions `conftest.py` to set up completions tests. For
example, setup_hf_text_completion_function allows for testing
text2text_generation and text_generation models using the same test
file.
- Common test code is now in pytest fixtures instead of common methods.
- For a number of the completion tests, I have broken out the asserts to
individual tests instead of running one giant test. This makes it easier
to identify regressions in specific patterns around invoking skills.
- Added retry logic to conversationSummarySkill
- renamed tests to be more descriptive.
@awharrison-28 awharrison-28 requested a review from dluc May 21, 2023 05:34
@github-actions github-actions bot added the python Pull requests for the Python Semantic Kernel label May 21, 2023
@dluc dluc enabled auto-merge (squash) May 21, 2023 06:02
@dluc dluc merged commit 00ff74e into main May 21, 2023
@alexchaomander alexchaomander self-requested a review May 21, 2023 16:11
shawncal pushed a commit to shawncal/semantic-kernel that referenced this pull request Jul 6, 2023
### Motivation and Context
Python integration tests were failing consistently for 2 reasons. First,
all ubuntu system tests were failing due to missing shared object files
related to cuda. Second, macOS + python 3.11 tests were failing due to
hsnwlib unable to install on native hardware.

See microsoft#1066 for
description of changes to integration test code to condense test code,
make better use of pytest fixtures, and reduce the number of calls to
external APIs (OpenAI, AOAI)

### Description
- Integration tests were failing on Linux due to a regression in python
torch=2.0.1. Downgrading the version to 2.0.0 resolved these failures.
- For MacOS + 3.11 specifically, setting the environment variable
`HNSWLIB_NO_NATIVE=1` ensures that the hnswlib wheel can be built and
installed properly on M1 hardware.
- Python integration tests reliably and consistently passing:
https://github.com/microsoft/semantic-kernel/actions/runs/5035748596
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants