Skip to content

Conversation

@sfc-gh-kbregula
Copy link
Contributor

@sfc-gh-kbregula sfc-gh-kbregula commented Mar 6, 2023

📚 Context

This PR makes two changes:

  • All Python versions are tested if needed.
  • The list of installed python dependencies is generated and stored.

Tested Python versions

Checking whether tests should be run for all supported Python versions, or whether it is enough to check the oldest and latest versions depending on what event triggered the current GitHub Action build to run.

  • For pull_request event, we test all supported Python versions when at least one of the conditions is met:

    • PR has "dev:full-matrix" label
    • Python dependencies have been modified

    In other case, we only test latest and oldest supported Python version

  • For push event, we return true when the default branch is checked. In other case, we only tests latest and oldest supported Python version

  • For other events, we only test latest and oldest supported Python version.

Constraint files

For each build, a list of all Python dependencies that have been installed is generated, written to a file, and published as an artifact. In addition, for successful builds on develop branch, these files are also published on constraints-develop branch. This way, we track a tested and working set of dependencies.

Those “known-to-be-working” constraints are generated per major/minor Python version for all supported Python versions.

To use the constraints file for installation, you need to add the -c flag to pip and pass the URL to the file:

PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
pip install streamlit -c "https://raw.githubusercontent.com/streamlit/streamlit/constraints-develop/constraints-${PYTHON_VERSION}.txt"

Next steps

As a next step, I would like to make the following changes:

Contribution License Agreement

By submitting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch 2 times, most recently from 68b183b to c7c0ead Compare March 6, 2023 12:26
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why my contribution doesn't work without it even though I didn't change anything in cache yarn, but now it's needed.

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 4185257 to b126a09 Compare March 6, 2023 14:09
@sfc-gh-kbregula sfc-gh-kbregula marked this pull request as ready for review March 6, 2023 14:10
@sfc-gh-tszerszen sfc-gh-tszerszen self-requested a review March 6, 2023 14:47
Copy link
Contributor

@sfc-gh-tszerszen sfc-gh-tszerszen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't wait to see this merged 😍

@sfc-gh-kbregula sfc-gh-kbregula added the do-not-merge PR is blocked from merging label Mar 6, 2023
Copy link
Collaborator

@vdonato vdonato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't have time to look at this today, but I'll take a pass tomorrow

Copy link
Collaborator

@vdonato vdonato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor nits, but LGTM

I think we may also want to run this by ProdSec since I'm not sure whether they'll have an opinion on automated commits being made to the repo

@sfc-gh-kbregula
Copy link
Contributor Author

I think we may also want to run this by ProdSec since I'm not sure whether they'll have an opinion on automated commits being made to the repo

I am working on it. Prodsec doesn't see the problem, but I have yet to confirm this with CorpSec, which manages the Github infra.

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 6a9bddc to 98ef641 Compare March 8, 2023 13:18
@sfc-gh-kbregula sfc-gh-kbregula removed the do-not-merge PR is blocked from merging label Mar 22, 2023
@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 1241261 to 022a3b7 Compare March 22, 2023 19:00
@sfc-gh-kbregula
Copy link
Contributor Author

I have approvals from the corp sec team and the prod sec team for this change. RA is not needed.

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 022a3b7 to 8b5ea08 Compare March 22, 2023 19:12
@sfc-gh-kbregula sfc-gh-kbregula added security-assessment-completed Security assessment has been completed for PR dev:full-matrix and removed dev:full-matrix labels Mar 22, 2023
@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from f8534fb to db216d9 Compare March 22, 2023 22:33
@tconkling
Copy link
Contributor

Thanks for taking this on, @sfc-gh-kbregula! A couple nits:

  1. I think informative PR descriptions are always important - but especially on PRs that change a lot of code and have an impact on workflows. (I'm not a fan of our PR template - it makes for very low signal:noise descriptions - and I advocate just using it as a guideline ("make sure you at least include XYZ bits of data"), instead of as a literal template.) For this particular PR, I'd love to see the template thrown out and replaced with a description of what changes are included, and why they are important to us, so that future readers can understand the context.

@sfc-gh-tteixeira writes amazing PR descriptions - see #5857 for the gold standard. (We definitely don't need all PR descriptions to be this thorough, but it's a good north star to aim for!)

  1. I think we're missing a lot of valuable documentation in the code itself. In particular,
  • the build_info.py script is a pretty big chunk of code, but it's not clear at a glance what it is doing and why it is useful. A docstring/comment at the top of the file saying "here's what this does and here's where it's used" would go a long way. I think it'd also be great to get docstrings for most of its functions. should_test_all_python_versions - what's its purpose, and when does it return True? What are the "output variables" in get_output_variables? Why do we save the environment variables we do, and who uses them?
  • The PR removes the set_python_versions/action.yml file, which had a short, simple, understandable description. There's a lot of new Github actions code (especially in python-versions.yml) that doesn't contain descriptions of what it's doing (and why), and as a result it's harder to follow.

We don't need to document literally everything we do - sometimes code really is simple and short enough that you can understand it without additional docstrings or comments. But in general, I prefer over-documenting to under-documenting; it makes life easier for coworkers, and also for the future versions of ourselves who may have forgotten the whats and whys of the code when we revisit it months or years later :)

Copy link
Contributor

@tconkling tconkling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Whoops, I left my feedback in a non-review comment - see above!)

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 8b85473 to 14d777a Compare March 29, 2023 20:01
@sfc-gh-kbregula
Copy link
Contributor Author

@tconkling Thanks for the review. I added more documentation and updated the PR description to add more context to this change.

Copy link
Contributor

@tconkling tconkling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation you added is fantastic. Thanks @sfc-gh-kbregula!

@sfc-gh-kbregula sfc-gh-kbregula force-pushed the sfc-gh-kbregula-constraints-public branch from 8e57404 to 69381b8 Compare April 4, 2023 15:34
@sfc-gh-mnowotka sfc-gh-mnowotka merged commit 8859424 into develop Apr 4, 2023
@mayagbarnes mayagbarnes mentioned this pull request Apr 6, 2023
1 task
@vdonato vdonato deleted the sfc-gh-kbregula-constraints-public branch November 2, 2023 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security-assessment-completed Security assessment has been completed for PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants