Skip to content

S3 migration - set s3 v2 provider as default#6827

Merged
bentsku merged 10 commits intov2from
s3-migration
Mar 23, 2023
Merged

S3 migration - set s3 v2 provider as default#6827
bentsku merged 10 commits intov2from
s3-migration

Conversation

@steffyP
Copy link
Member

@steffyP steffyP commented Sep 6, 2022

Summary

This PR should remain minimal in terms of code changes, and is mostly used for keeping track of the PRs regarding the new s3 provider going directly into master and for tracking test failure rates against the new provider.

PRs

The following PRs have been created to fully implement the new s3 provider:

Tasks

  • Migrate all s3 tests to pytest and validates them against AWS

  • Create the new ASF provider

  • Allow patching of botocore specs to match actual AWS behaviour

  • Reroute virtual host addressed requests as path addressed ones to take advantage of the builtin ASF routing

  • Implement major S3 features:

    • S3 notifications
    • All major basic OPs (GetObject, GetBucket, etc)
    • Pre-signed URLs
    • Pre-signed POST
    • S3 Static Website Hosting
    • CORS
  • Fix ongoing issues with current provider not tackled in above major features

    • VersionId and Expires always present
    • GetObjectAttributes not implemented
    • Empty tagging lists
    • Delete versioned objects
    • Fix Terraform tests (fixed in S3 asf fixes detected by Terraform test suite #7027 + full test run green here)
    • remove legacy VirtualHostRewriter
    • Fix issues with presigned urls
  • Implement the Pro provider

  • Implement S3 ASF test job for CircleCI

@steffyP steffyP temporarily deployed to localstack-ext-tests September 6, 2022 12:11 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests September 6, 2022 12:37 Inactive
@github-actions
Copy link

github-actions bot commented Sep 6, 2022

LocalStack integration with Pro

       1 files   -     2         1 suites   - 2   1h 7m 29s ⏱️ - 25m 55s
1 821 tests  -   16  1 478 ✔️ +  29  342 💤  -   46  1 +1 
1 821 runs   - 752  1 478 ✔️  - 338  342 💤  - 415  1 +1 

For more details on these failures, see this check.

Results for commit b0af983. ± Comparison against base commit 6e81652.

♻️ This comment has been updated with latest results.

@bentsku bentsku mentioned this pull request Sep 6, 2022
@steffyP steffyP closed this Sep 6, 2022
@bentsku bentsku reopened this Sep 6, 2022
@bentsku bentsku temporarily deployed to localstack-ext-tests September 6, 2022 17:24 Inactive
@bentsku
Copy link
Contributor

bentsku commented Sep 7, 2022

Note: we have 53 failed tests for now, to keep track of progress.

@steffyP steffyP temporarily deployed to localstack-ext-tests September 8, 2022 16:13 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests September 9, 2022 15:06 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests September 12, 2022 12:32 Inactive
@bentsku
Copy link
Contributor

bentsku commented Sep 13, 2022

We are at 92 tests failing for this branch before merging any functionality.

@bentsku bentsku temporarily deployed to localstack-ext-tests September 13, 2022 23:06 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests September 14, 2022 14:55 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests September 14, 2022 20:38 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests September 19, 2022 16:13 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests September 19, 2022 17:38 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests September 20, 2022 20:16 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests October 24, 2022 15:58 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests October 25, 2022 09:37 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests October 25, 2022 11:30 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests October 25, 2022 15:14 Inactive
@steffyP steffyP temporarily deployed to localstack-ext-tests October 31, 2022 12:48 Inactive
@bentsku bentsku temporarily deployed to localstack-ext-tests November 25, 2022 15:21 Inactive
@bentsku bentsku changed the title [Draft] S3 migration S3 migration - set s3 v2 provider as default Mar 20, 2023
@bentsku bentsku changed the base branch from master to v2 March 20, 2023 16:45
@bentsku bentsku temporarily deployed to localstack-ext-tests March 20, 2023 16:45 — with GitHub Actions Inactive
@bentsku bentsku marked this pull request as ready for review March 20, 2023 16:46
@bentsku bentsku requested a review from thrau as a code owner March 20, 2023 16:46
@bentsku bentsku self-requested a review March 20, 2023 16:46
@bentsku bentsku temporarily deployed to localstack-ext-tests March 20, 2023 18:16 — with GitHub Actions Inactive
@thrau thrau requested a review from macnev2013 as a code owner March 21, 2023 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants