Azure Storage: Azure storage support #3253 by sfali · Pull Request #3254 · akka/alpakka

sfali · 2024-08-27T13:20:21Z

References #3253

Work in progress, still needs to add more tests and documentations.

Code style is copied from AWS S3 connector.

It parses the returned XML and populate StorageException

1. Populates required date and version header 2. Reads current request and create authorization token as per AWS requirements 3. Populates authorization header

Current implementation only supports Blob and File services. Currently supported functions: 1. Get blob and file 2. Put blob 3. Create file 3. Put range 4. Clear range 5. Delete blob and file 6. Get blob and file properties

…kka#3253

1. Create query string with SAS token 2. Initialize query string

…3253

…/StorageAttributes.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

…/impl/AzureStorageStream.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

…/models.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

girdharshubham

Looking good so far.

…/StorageAttributes.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

1. Make sure we don't emit empty string 2. correct configuration

…ure_storage

1. Request builder for delete container 2. Implementation of delete container and Scala and Java API 3. Tests for delete container 4. Documentation

1. Request builder for create & delete directories 2. Implementation of create & delete directories and Scala and Java API 3. Tests for create & delete directories 4. Documentation

sfali · 2024-09-14T18:17:08Z

Issue with Java API was misconfiguration, not sure why AzuriteIntegrationSpec is failing, any idea @girdharshubham?

girdharshubham · 2024-09-24T00:16:10Z

Issue with Java API was misconfiguration, not sure why AzuriteIntegrationSpec is failing, any idea @girdharshubham?

Hi @sfali
Sorry this got delayed, I will check this soon.

sebastian-alfers · 2024-09-30T10:31:39Z

Hi @sfali, can you share and update on this PR? Do you want to see your contribution merged for the next release?

sfali · 2024-09-30T14:06:24Z

Hi @sfali, can you share and update on this PR? Do you want to see your contribution merged for the next release?

What do you want me to share @sebastian-alfers? The PR is failing due to couchbase and ironmq connectors are failing. not sure what should I do with these connectors.

sebastian-alfers · 2024-09-30T14:54:57Z

We fixed both in main, so feel free to rebase and give it a try.

sfali · 2024-09-30T17:11:44Z

We fixed both in main, so feel free to rebase and give it a try.

@sebastian-alfers Now scaaldoc and jms connector failed. Can pin point what's going? Can you please take a look and let me know what's going on.

Thanks

sebastian-alfers · 2024-10-01T13:47:54Z

Thank you for bringing this to our attention. The scaladoc failure can be ignored for now, and the jms was something we just fixed in main, so please rebase again.

sfali · 2024-10-01T15:24:36Z

Thank you for bringing this to our attention. The scaladoc failure can be ignored for now, and the jms was something we just fixed in main, so please rebase again.

Thanks @sebastian-alfers , Everything is passing other than scaaldoc

sebastian-alfers · 2024-10-02T06:30:28Z

Please let me know once you want to merge it.

sfali · 2024-10-02T16:10:21Z

Please let me know once you want to merge it.

This can be merge anytime. It needs to be approved.

sfali · 2024-10-03T13:33:04Z

It seems scaladoc pipeline needs to be successful before it can be merged. @sebastian-alfers

Susmit07 · 2024-11-07T18:09:00Z

+                  objectPath = objectPath,
+                  requestBuilder = requestBuilder)
+
+  private[storage] def putBlob(objectPath: String,


@sfali one question for large files may be of > 5-10Gb .. S3 has multipartUploadAPI for azure blob how are we going to achieve it ?

splitting in chunks?

adding some UTs would have been helpful..

sfali added 30 commits August 23, 2024 16:55

Setting up module for Azure storage akka#3253

06eac15

Azure storage common settings akka#3253

42b8505

Some common functions akka#3253

4412dec

Supporting model classes akka#3253

dc26cea

Exception representing returned error akka#3253

882afcd

It parses the returned XML and populate StorageException

Helper class to sign and populate Authorization header akka#3253

5a8728e

1. Populates required date and version header 2. Reads current request and create authorization token as per AWS requirements 3. Populates authorization header

Implementation of Azure storage functions akka#3253

3616c5f

Current implementation only supports Blob and File services. Currently supported functions: 1. Get blob and file 2. Put blob 3. Create file 3. Put range 4. Clear range 5. Delete blob and file 6. Get blob and file properties

Scala and Java facade to implementation akka#3253

7626390

Introducing Akka stream attributes in order to configure http request a…

f1de209

…kka#3253

Resolve settings from Akka stream attributes akka#3253

426b800

Added support for SharedKeyLite authorization akka#3253

e41869e

Added support for SAS authorization akka#3253

5d3b55d

use option for null String akka#3253

dcc9c21

function to get default settings akka#3253

582edb3

Support for SAS authorization akka#3253

7dd77c6

1. Create query string with SAS token 2. Initialize query string

Splitting service operations based on service type akka#3253

d29f3f9

Splitting service operations based on service type akka#3253

997746a

StorageException test akka#3253

34832f6

creating constants for authorization types akka#3253

6cc3254

implicit class to handle validations akka#3253

d51c06a

validations of configs and falling back to default values akka#3253

7b3ed57

StorageSettings tests akka#3253

07e64a5

support for create container operation akka#3253

44c2f84

support for endpoint URL for local testing akka#3253

93c2aab

using end point if defined to create URI akka#3253

7ac0aa4

move signer out of retry flow, we don't need to sign every time akka#…

9f013cb

…3253

copy right header akka#3253

9b81f84

pass implicit ActorSystem akka#3253

3ad11ea

parse content MD5 as well akka#3253

790fd5a

populate environment variables to run test akka#3253

d140558

sfali and others added 3 commits September 9, 2024 11:03

Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…

4cfe27a

…/StorageAttributes.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…

f5971a1

…/impl/AzureStorageStream.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…

1eaf5df

…/models.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

girdharshubham reviewed Sep 9, 2024

View reviewed changes

Comment thread azure-storage/src/main/scala/akka/stream/alpakka/azure/storage/StorageAttributes.scala

Comment thread azure-storage/src/main/scala/akka/stream/alpakka/azure/storage/StorageAttributes.scala

girdharshubham reviewed Sep 9, 2024

View reviewed changes

sfali and others added 6 commits September 10, 2024 10:45

Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…

18f4336

…/StorageAttributes.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…

fcd4326

…/StorageAttributes.scala Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>

Fail safe error message akka#3253

948db31

1. Make sure we don't emit empty string 2. correct configuration

Merge remote-tracking branch 'origin/3253_azure_storage' into 3253_az…

60f9ae9

…ure_storage

Delete Container support akka#3253

689cd53

1. Request builder for delete container 2. Implementation of delete container and Scala and Java API 3. Tests for delete container 4. Documentation

Create & Delete directory support akka#3253

d6d9ed0

1. Request builder for create & delete directories 2. Implementation of create & delete directories and Scala and Java API 3. Tests for create & delete directories 4. Documentation

sfali added 2 commits September 30, 2024 12:04

Merge branch 'master' into 3253_azure_storage

1c9c3cf

fix import akka#3253

43f347f

Merge branch 'master' into 3253_azure_storage

1e07711

sebastian-alfers approved these changes Oct 3, 2024

View reviewed changes

sebastian-alfers merged commit 4049dca into akka:main Oct 3, 2024

sfali deleted the 3253_azure_storage branch October 3, 2024 17:37

Susmit07 reviewed Nov 7, 2024

View reviewed changes

sebastian-alfers mentioned this pull request Mar 11, 2025

Alpakka connector for Azure Storage #3253

Closed

Conversation

sfali commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

girdharshubham left a comment

Choose a reason for hiding this comment

Uh oh!

sfali commented Sep 14, 2024

Uh oh!

girdharshubham commented Sep 24, 2024

Uh oh!

sebastian-alfers commented Sep 30, 2024

Uh oh!

sfali commented Sep 30, 2024

Uh oh!

sebastian-alfers commented Sep 30, 2024

Uh oh!

sfali commented Sep 30, 2024

Uh oh!

sebastian-alfers commented Oct 1, 2024

Uh oh!

sfali commented Oct 1, 2024

Uh oh!

sebastian-alfers commented Oct 2, 2024

Uh oh!

sfali commented Oct 2, 2024

Uh oh!

sfali commented Oct 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Susmit07 Nov 7, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sfali commented Aug 27, 2024 •

edited

Loading

sfali commented Oct 3, 2024 •

edited

Loading