[Kaniko] Add S3/MinIO Build Context Support and Improve Builder Environment Handling #8849

adamdelman · 2025-11-04T22:47:07Z

📝 Description

Extend Kaniko build support to handle S3 build-context sources and improve how builder environment variables are generated.
Also standardize logging calls in builder flows and ensure Kubernetes pod environments are always initialized as typed lists.

🛠️ Changes Made

Kubernetes Helper

Initialize Pod.env as a typed list (list[V1EnvVar]) instead of None to ensure safe and consistent mutation.

Builder

Introduce S3 context detection:
- Add is_s3_source flag.
- Refine source-scheme logic to treat s3:// similarly to local/v3io contexts.
Integrate new _enrich_kaniko_env_for_s3_context():
- When the build context is s3://, set AWS_REGION using:
  - existing env vars → or process env → or fallback "default-region".
- Only if no real region is found (fallback case), also add S3_FORCE_PATH_STYLE=true for MinIO compatibility.
Replace all mlrun.utils.logger.* calls with mlrun.utils.helpers.logger.* for consistent structured logging.
Update _generate_builder_env():
- Accept list-style builder env ([{name, value}]) instead of a dict.
- Convert entries into V1EnvVar objects.
- Merge with project secrets, avoiding duplicates based on the env name.

🧪 Testing

Verified build flows with:
- s3:// → correct region inference and MinIO path-style injection.
- v3io://, local paths, and http:// → unchanged behavior.
Confirmed builder env generation correctly merges user env and project secrets.
Ensured updated logging source prints structured log fields as expected.

🔗 References

CEML-520

🚨 Breaking Changes?

No

🔍 Additional Notes

Improves robustness for S3/MinIO user workflows without requiring extra configuration.
Eliminates inconsistencies around builder environment mutation and logging.

- Enable S3 build context in Kaniko: - Add _enrich_kaniko_env_for_s3_context() to auto-set AWS_REGION=us-east-1 and S3_FORCE_PATH_STYLE=true when using s3:// - Pass builder env vars to Kaniko pod and convert AWS creds to secrets - K8s pod refactor: - Rename BasePod → Pod - Maintain explicit env list and add add_env_var() - Implement replace_env_vars_with_secrets() and _create_secret_for_env_var() to materialize sensitive env vars as K8s secrets - Builder / build flow improvements: - Accept env from request or function spec (`builder_env` fallback to `spec.env`) - Normalize builder env handling and logging via mlrun.utils.helpers.logger - Call function.try_auto_mount_based_on_config() before deploy - Treat s3://, v3io://, http(s):// sources distinctly; mount/copy as needed - Generate project secret-backed envs without overriding provided vars https://iguazio.atlassian.net/browse/CEML-520

liranbg

The PR incorporate few changes at once

Fix for s3 region/force path
Env to secrets
Access key handling

This PR needs to be broken into smaller prs, each has its own risks and need to be carefully taken care of. for the (2) I am not sure I udnerstand the reason behind it. for (3) it seems to be incomplete and also, why is it needed? and for (1) - I think the safest way to proceed is to check if we can already can overcome the issue by providing explicit kaniko build args.

liranbg · 2025-11-05T07:42:45Z

server/py/framework/utils/singletons/k8s.py

+    def add_env_var(self, env_var: client.V1EnvVar):
+        self.env.append(env_var)


the operation is not idempotent and prone to duplicate envvars

Neither is add_volume, why do we need it to be idempotent?

liranbg · 2025-11-05T07:43:28Z

server/py/framework/utils/singletons/k8s.py

+                secret_env_var = self._create_secret_for_env_var(
+                    name=env_var.name,
+                    value=env_var.value,
+                    k8s_helper=k8s_helper,
+                )
+                self.env[index] = secret_env_var


what component is responsible to remove those secret later on? if I have 100 envvars, will it create 100 secrets? what's the reasoning behind it?

This exactly the same as we do in mlrun_pipelines.common.ops._replace_env_vars_with_secrets, which was added in this PR you approved, where we create obfuscate secret env vars for each pipeline run.
The reasoning is that we do not want AWS/V3IO credentials to be visible on pod YAML's.

liranbg · 2025-11-05T07:44:21Z

server/py/framework/utils/singletons/k8s.py

+        secret_name, _ = k8s_helper.store_auth_secret(
+            username=name,
+            access_key=value,
+        )


does that aligned with igz4/oauth/igz3 expectations?

Again, this is the same as mlrun_pipelines.common.ops._replace_env_vars_with_secrets

liranbg · 2025-11-05T07:46:27Z

server/py/services/api/utils/builder.py

+                name="AWS_REGION",
+                value="us-east-1",


why defaulting to this region?

Because kaniko expects there to be a region set, even for minio, and this is the default AWS region.

Because kaniko expects there to be a region set, even for minio, and this is the default AWS region.

not 100%, on aws - admin can set its default region. I suggest to not default ourselves by either get it using aws (perhpas using aws command on init container / mlrun configuration) but def not hardcoded

adamdelman · 2025-11-05T12:05:55Z

The PR incorporate few changes at once

Fix for s3 region/force path

Env to secrets

Access key handling

This PR needs to be broken into smaller prs, each has its own risks and need to be carefully taken care of. for the (2) I am not sure I udnerstand the reason behind it. for (3) it seems to be incomplete and also, why is it needed? and for (1) - I think the safest way to proceed is to check if we can already can overcome the issue by providing explicit kaniko build args.

@liranbg
The reason for adding item number 2 is that we are adding sensitive credentials to the kaniko pod, and I don't want the to be visible in the pod yaml.
Item number 3 was not really added, if you look carefully it was already there , in this code block:

        "V3IO_ACCESS_KEY", auth_info.data_session or auth_info.access_key
    )
    username = builder_env.get("V3IO_USERNAME", auth_info.username)```
 The only difference is that now I also obfuscate these if they exist.

server/py/services/api/utils/builder.py

liranbg · 2025-11-13T06:37:34Z

server/py/services/api/utils/builder.py

+                name="AWS_REGION",
+                value="us-east-1",


Because kaniko expects there to be a region set, even for minio, and this is the default AWS region.

not 100%, on aws - admin can set its default region. I suggest to not default ourselves by either get it using aws (perhpas using aws command on init container / mlrun configuration) but def not hardcoded

liranbg · 2025-11-13T06:41:09Z

server/py/services/api/utils/builder.py

+
+        # MinIO requires path-style URLs
+        already_has_path_style = any(
+            env.name == "S3_FORCE_PATH_STYLE" and env.value for env in env_vars


It is not 100% safe for "new" aws regions not supporting this ENV/configuration as they require virtual-host path. maybe we can force this configuration when we deploy against minio (mlrun ce-like configuration) - but not when working natively against s3

liranbg

looking better, worth adding some ut use cases coverage to cover conventional flows

…er Environment Handling" (#9052) Reverts #8849 Due to 1. https://iguazio.atlassian.net/browse/ML-11743 2. https://iguazio.atlassian.net/browse/ML-11731 Overall, there are few issues with this PR as it enriches kaniko pod with s3 information and ignoring project secrets. PR needs to be rewritten and intensively covered.

adamdelman added 16 commits August 13, 2025 13:13

Merge branch 'development' of github.com:mlrun/mlrun into development

ed9c51d

Merge branch 'development' of github.com:mlrun/mlrun into development

efaac7c

Merge branch 'development' of github.com:mlrun/mlrun into development

7a23a19

Merge branch 'development' of github.com:mlrun/mlrun into development

1e6f681

Merge branch 'development' of github.com:mlrun/mlrun into development

fad418b

Merge branch 'development' of github.com:mlrun/mlrun into development

7afe994

Merge branch 'development' of github.com:mlrun/mlrun into development

19855b5

Merge branch 'development' of github.com:mlrun/mlrun into development

280080f

Merge branch 'development' of github.com:mlrun/mlrun into development

5a14081

Merge branch 'development' of github.com:mlrun/mlrun into development

63f7b03

Merge branch 'development' of github.com:mlrun/mlrun into development

3ea2c72

Merge branch 'development' of github.com:mlrun/mlrun into development

74d0595

Merge branch 'development' of github.com:mlrun/mlrun into development

16a24f7

Merge branch 'development' of github.com:mlrun/mlrun into development

23e4e8f

Merge branch 'development' of github.com:mlrun/mlrun into development

75b7375

Merge branch 'development' of github.com:mlrun/mlrun into development

9608427

github-actions bot added PR title format area/sdk area/server area/api area/framework area/tests labels Nov 4, 2025

adamdelman force-pushed the adamd/CEML-520 branch from 99dfa98 to ca30161 Compare November 4, 2025 22:47

adamdelman force-pushed the adamd/CEML-520 branch from 908a04f to 494998f Compare November 4, 2025 22:56

adamdelman changed the title ~~Adamd/ceml 520~~ [Kaniko] add S3 Kaniko context + secret env; refactor BasePod→Pod Nov 4, 2025

adamdelman marked this pull request as ready for review November 4, 2025 22:59

adamdelman requested review from a team and quaark as code owners November 4, 2025 22:59

github-actions bot removed the PR title format label Nov 4, 2025

adamdelman added 2 commits November 5, 2025 07:23

fix

f381189

Merge branch 'development' of github.com:mlrun/mlrun into adamd/CEML-520

47893ac

adamdelman force-pushed the adamd/CEML-520 branch from e330044 to 47893ac Compare November 5, 2025 05:25

liranbg requested changes Nov 5, 2025

View reviewed changes

adamdelman added 2 commits November 6, 2025 15:53

Merge branch 'development' of github.com:mlrun/mlrun into adamd/CEML-520

3784c01

fix

0ae6b45

adamdelman changed the title ~~[Kaniko] add S3 Kaniko context + secret env; refactor BasePod→Pod~~ [Kaniko] Add S3/MinIO Build Context Support and Improve Builder Environment Handling Nov 6, 2025

adamdelman requested a review from liranbg November 9, 2025 08:07

adamdelman self-assigned this Nov 9, 2025

liranbg requested changes Nov 13, 2025

View reviewed changes

adamdelman added 3 commits November 18, 2025 11:22

fix

e305789

fix

5679037

fix

552fd71

adamdelman requested a review from liranbg November 26, 2025 13:29

liranbg requested changes Nov 27, 2025

View reviewed changes

adamdelman added 7 commits November 30, 2025 17:28

add tests

95b2455

fix

8e3c4c6

fix

9a41913

fix

cc9fbc3

fix

671864f

pr fixes

cab858a

pr fixes

3c8e0d4

adamdelman requested a review from liranbg December 1, 2025 13:40

liranbg approved these changes Dec 2, 2025

View reviewed changes

liranbg merged commit 8370daa into mlrun:development Dec 2, 2025
13 checks passed

adamdelman deleted the adamd/CEML-520 branch December 2, 2025 12:35

liranbg mentioned this pull request Dec 14, 2025

Revert "[Kaniko] Add S3/MinIO Build Context Support and Improve Builder Environment Handling" #9052

Merged

		def add_env_var(self, env_var: client.V1EnvVar):
		self.env.append(env_var)

[Kaniko] Add S3/MinIO Build Context Support and Improve Builder Environment Handling #8849

[Kaniko] Add S3/MinIO Build Context Support and Improve Builder Environment Handling #8849

Conversation

adamdelman commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description

🛠️ Changes Made

Kubernetes Helper

Builder

🧪 Testing

🔗 References

🚨 Breaking Changes?

🔍 Additional Notes

Uh oh!

liranbg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamdelman Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamdelman commented Nov 5, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liranbg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adamdelman commented Nov 4, 2025 •

edited

Loading

adamdelman Nov 5, 2025 •

edited

Loading