Skip to content

Implement general awsEndpoint + lambda integration#428

Closed
EikeKohl wants to merge 2 commits intohuggingface:mainfrom
EikeKohl:allow-aws-lambda-endpoint
Closed

Implement general awsEndpoint + lambda integration#428
EikeKohl wants to merge 2 commits intohuggingface:mainfrom
EikeKohl:allow-aws-lambda-endpoint

Conversation

@EikeKohl
Copy link
Copy Markdown

@EikeKohl EikeKohl commented Sep 5, 2023

This small refactoring PR implements a general awsEndpoint, enabeling the user to not only connect sagemaker endpoints, but also lambda function URLs that require AWS_IAM authentication.
I had to implement this change for my personal setup, because upon requesting an LLM generation, the lambda performs additional scaling logic to provision / delete the sagemaker endpoint to save LLM hosting costs. I think that everyone could benefit from this functionality.

@nsarrazin
Copy link
Copy Markdown
Contributor

Hello and thanks for contributing!

On paper this looks good to me, I haven't tested yet but what do you think @philschmid ?

@philschmid
Copy link
Copy Markdown

Hello and thanks for contributing!

On paper this looks good to me, I haven't tested yet but what do you think @philschmid ?

Agree! Looks good, but we need to make sure both ways work and should have some checks that people will not provide, e.g. ecs or something as service.

@EikeKohl
Copy link
Copy Markdown
Author

EikeKohl commented Sep 8, 2023

Thank you for considering my contribution 🙏. I actually tested it with both, Lambda Function URL and Sagemaker invocation URL and it worked 🥳. @philschmid If ecs is specified, the build will fail with a validation error from Zod, because I defined the service parameter to be a Union of the Literals lambda and sagemaker.

P.S. I noticed the workflow failed with a linting error, so I pushed another commit after I ran npm run format to hopefully fix it.

@nsarrazin nsarrazin self-requested a review October 2, 2023 08:45
@nsarrazin nsarrazin self-assigned this Oct 2, 2023
@nsarrazin
Copy link
Copy Markdown
Contributor

Hi @EikeKohl, sorry for taking so long to take a look at this PR. I've added support for your changes in this PR with more modular backends for the future.

Can you take a look and let me know if this works with your lambda workflow for now?

@EikeKohl
Copy link
Copy Markdown
Author

EikeKohl commented Nov 2, 2023

Hi @EikeKohl, sorry for taking so long to take a look at this PR. I've added support for your changes in this PR with more modular backends for the future.

Can you take a look and let me know if this works with your lambda workflow for now?

Looks good to me and works for my workflow 🙂 Thank you!

@nsarrazin
Copy link
Copy Markdown
Contributor

Cool, going to close this for now then, as we'll merge the other PR soon. I'll ping you when it's done!

@nsarrazin nsarrazin closed this Nov 2, 2023
@nsarrazin nsarrazin removed their assignment Nov 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants