Skip to content

Add weekly automated check for outdated third-party test vectors#2933

Merged
sgmenda merged 11 commits intoaws:mainfrom
sgmenda:update-vectors-ci
Jan 23, 2026
Merged

Add weekly automated check for outdated third-party test vectors#2933
sgmenda merged 11 commits intoaws:mainfrom
sgmenda:update-vectors-ci

Conversation

@sgmenda
Copy link
Copy Markdown
Contributor

@sgmenda sgmenda commented Jan 8, 2026

Issues:

We currently rely on manual updates of test vectors from third-party sources like Wycheproof. This PR automates the detection of outdated vectors.

Description of changes:

Adds a GitHub Actions workflow that runs weekly on Wednesdays at 10 AM PT to check if third-party test vectors need updating. When outdated vectors are detected, the workflow creates a GitHub issue with update instructions using a new issue template.

The workflow:

  • Runs sync.py --check to detect outdated vectors without modifying files
  • Checks for open issues with the "vectors" label
  • If no issues, then creates an issue with the "vectors" label

The issue template provides step-by-step instructions for updating vectors locally, including running tests and regenerating build files.

Call-outs:

  • This only works for vectors onboarded to the third_party/vectors system
  • The workflow detects and notifies but does not automatically update vectors
  • It only checks vectors we actually use, not all available Wycheproof vectors
  • fixes a bug in sync.py --check

Testing:

The sync.py --check command was tested locally and confirmed to work correctly. The workflow can be manually triggered to debug any issues. The deduplication logic follows GitHub's recommended pattern for scheduled issue creation.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

@sgmenda sgmenda self-assigned this Jan 8, 2026
@sgmenda sgmenda requested a review from a team as a code owner January 8, 2026 18:44
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jan 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.06%. Comparing base (bee13c3) to head (552f85a).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2933      +/-   ##
==========================================
- Coverage   78.08%   78.06%   -0.02%     
==========================================
  Files         682      682              
  Lines      118355   118355              
  Branches    16607    16610       +3     
==========================================
- Hits        92419    92397      -22     
- Misses      25048    25070      +22     
  Partials      888      888              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sgmenda sgmenda marked this pull request as draft January 22, 2026 18:40
@sgmenda sgmenda marked this pull request as ready for review January 22, 2026 19:08
@sgmenda sgmenda merged commit 83d9de2 into aws:main Jan 23, 2026
399 of 400 checks passed
@sgmenda sgmenda deleted the update-vectors-ci branch January 23, 2026 22:42
@justsmth justsmth mentioned this pull request Feb 20, 2026
justsmth added a commit that referenced this pull request Feb 20, 2026
### Description of changes: 
* Bump urllib3 from 2.6.0 to 2.6.3 in /tests/ci by @dependabot[bot] in
#2932
* Add weekly automated check for outdated third-party test vectors by
@sgmenda in #2933
* Enable Hybrid PQ KeyShares by default by @alexw91 in
#2531
* Remove AVX conditional from cmake script by @torben-hansen in
#2958
* openssl-ca command implementation for self-sign certificates by
@skmcgrail in #2937
* Initial Framework for Using Doxygen to Document Public Header Files by
@m271828 in #2908
* Move md4 out of FIPS module by @torben-hansen in
#2956
* Fix image-build-windows workflow to only push on workflow_call and
workflow_dispatch by @skmcgrail in
#2961
* Remove FIPS counter framework and other tidying up by @torben-hansen
in #2947
* Model Device Farm CI Resources in CDK by @skmcgrail in
#2965
* Adds a new randomness generation API by @torben-hansen in
#2963
* Migrate Android Testing to GitHub Actions by @skmcgrail in
#2969
* Ensure pkcs7 checks ASN1_TYPE->type by @skmcgrail in
#2968
* Fix checkout logic for android-omnibus by @skmcgrail in
#2970
* Add missing env vars to check-vectors workflow step by @sgmenda in
#2962
* Shorten Windows Build Directory Path by @skmcgrail in
#2974
* Bump mysql cluster version by @WillChilds-Klein in
#2967
* Integrate Wycheproof ML-DSA test vectors by @sgmenda in
#2973
* Simplify FIPS conditional in top-level build script by @torben-hansen
in #2976
* Fix aws-lc-rs CI job by @justsmth in
#2966
* Add method to get type of ML-DSA instance configured under EVP PKEY by
@torben-hansen in #2980
* Nmap build needs liblinear by @justsmth in
#2985
* Disable SLP vectorizer for FIPS shared library builds on GCC 14+ by
@geedo0 in #2977
* Update Wycheproof ECDSA test vectors and fix workflow typo by @sgmenda
in #2972
* Address some CMake findings by @skmcgrail in
#2979
* Bump bytes from 1.7.1 to 1.11.1 in /tests/ci/lambda by
@dependabot[bot] in #2983
* Support GCC 4.8 for aarch64 by @justsmth in
#2964
* Free potential memory before assigning new pointer by @torben-hansen
in #2989
* Add PyOpenSSL integration test by @WillChilds-Klein in
#2992
* Ensure index argument is not negative in ASN1_BIT_STRING_set_bit by
@torben-hansen in #2987
* Ensure no overflow in signed output length in do_buf by @torben-hansen
in #2988
* Remove redundant CPython 3.9 integration test by @WillChilds-Klein in
#2996
* Ensure public key is set before verifying through ML-DSA verify by
@torben-hansen in #2990
* Correct CCM nids in object definition by @torben-hansen in
#2991
* Address Reported Bug Findings by @skmcgrail in
#3000
* Fix CI: gcc-4.8 by @justsmth in
#3011
* Fix Windows CI: use `cd /d` in run_windows_tests.bat to handle
cross-drive paths by @justsmth in
#3012
* Fix OPENSSL_memchr per C23 by @justsmth in
#3008
* Fix argument order in `hmac_copy` by @justsmth in
#3014
* Miscellaneous CI improvements by @skmcgrail in
#2978
* Fix CI: mariadb by @justsmth in
#3015
* Update Ubuntu 24:04 image compiler verification by @skmcgrail in
#3017
* Support WASM/Emscripten by @justsmth in
#2959
* Generate Rust Bindings by @justsmth in
#2999

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and the ISC license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants