Switch to using gcsfuse profile feature in aiml gcs-bucket mounts in slurm cluster blueprints by gargnitingoogle · Pull Request #5047 · GoogleCloudPlatform/cluster-toolkit

gargnitingoogle · 2025-12-30T11:55:30Z

Description

This PR uses GCSFuse profile parameter to replace the more fine-grained cache and config control parameters in the slurm cluster blueprints for A3-ultra, A4x-high, and A4-high node types for all the aiml workload bucket mount options. This makes the mount options in blueprints shorter and easier to maintain, as GCSFuse automatically sets the optimal value for the parameter based on the profile value (details).
This also replaces - with _ in the gcsfuse mount_options in the changed blueprints, as these mount-options are copied directly to /etc/fstab file for mount, and it's required to use _ instead of - in gcsfuse mount-options in such a case (persistent mounting documentation)
Manual testing done
- Launched and tested the gcsfuse mounts in a slurm cluster for machine-type a3-ultragpu-8g, and it reflected in the flags applied correctly on a compute node.
- Launched and tested the gcsfuse mount in a slurm cluster for machine-type a4-highgpu-8g, and it reflected in the mount logs and the file operations correctly in the mounted directories on the compute node.
- Testing pending on A4X

Note: Found couple of unrelated issues.

Got error Mounting failed as local mount: /gcs-checkpoints was already in use in fstab seems to indicate that the srun command causes the mounts to be added to /etc/fstab again, which fails as expected. Not sure if this is a bug or is expected.
During mv operations in the gcsfuse mounts in the slurm compute node, I got errors mv: preserving times for '/gcs/sample_1GB_renamed.txt': Operation not permitted and mv: preserving permissions for ‘/gcs/sample_1GB_renamed.txt’: Operation not permitted which are expected errors because gcsfuse mount isn't fully POSIX-compliant and doesn't support file permissions changes, or preserving/propagation system times other than the modification time. But these errors showed up here because mv command tries to do both these operations and they fail. The mv command itself completes with exit-code 0 as expected.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

Fork your PR branch from the Toolkit "develop" branch (not main) - DONE
Test all changes with pre-commit in a local branch # - Not sure what do here.
Confirm that "make tests" passes all tests - DONE
Add or modify unit tests to cover code changes - NOT APPLICABLE
Ensure that unit test coverage remains above 80% - NOT APPLICABLE
Update all applicable documentation - NOT APPLICABLE
Follow Cluster Toolkit Contribution guidelines # - DONE

gemini-code-assist · 2025-12-30T11:55:48Z

Summary of Changes

Hello @gargnitingoogle, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the A3 Mega GPU Slurm blueprint by integrating specialized cloud storage solutions tailored for machine learning workloads. It introduces dedicated GCS buckets for managing checkpoints, training data, and model serving, each configured for optimal performance and observability. Additionally, the blueprint now includes an automated step to install Go, likely supporting new tooling or services, and incorporates a minor cleanup in the underlying Terraform module for cloud storage buckets.

Highlights

New AI/ML Storage Buckets: Introduced dedicated Google Cloud Storage (GCS) buckets for checkpoints, training data, and model serving within the A3 Mega GPU blueprint. Each bucket is configured with specific GCSFuse profiles and enhanced logging options.
Go Installation Added: A new shell runner has been integrated into the deployment process of the A3 Mega GPU blueprint to automatically install Go 1.24.11 and update system PATH variables.
Enhanced GCSFuse Logging: All GCSFuse mount options across the blueprint have been updated to include detailed logging parameters (log_severity, log_file, log_format) for improved debugging and monitoring capabilities.
Terraform Module Refinement: A minor adjustment was made in the cloud-storage-bucket Terraform module, specifically commenting out a redundant line within the anywhere_cache precondition logic.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces several new Cloud Storage buckets to the a3mega-slurm-blueprint.yaml example, tailored for different machine learning workflow stages (checkpoints, training data, model serving). It also adds a startup script to install Go. My review focuses on improving the maintainability of the new script, cleaning up redundant configuration, and removing leftover debugging code. Additionally, please note that the pull request description is the default template and should be updated to explain the purpose of these changes and how they were tested, as per the repository's contribution guidelines (lines 42-44).

examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml

modules/file-system/cloud-storage-bucket/main.tf

gargnitingoogle · 2026-01-02T11:35:43Z

/gemini summary

gemini-code-assist · 2026-01-02T11:36:07Z

Summary of Changes

This pull request refines the A3 Ultra GPU Slurm blueprint by simplifying and standardizing the GCSFuse mount configurations for critical AI/ML workloads. It introduces dedicated GCSFuse profiles for checkpointing, training data, and model serving, replacing a multitude of explicit caching and metadata options with more concise and potentially optimized settings. This change enhances clarity and maintainability of the storage configurations.

Highlights

GCSFuse Profile Adoption: The GCSFuse mount options for checkpointing, training data, and model serving have been updated to utilize specialized profile settings (e.g., profile=aiml-checkpointing, profile=aiml-training, profile=aiml-serving).
Simplified Mount Options: Verbose and explicit GCSFuse options related to caching, metadata handling, and streaming writes (such as implicit-dirs, metadata-cache-*, stat-cache-*, type-cache-*, file-cache-*, enable-streaming-writes) have been removed, as their functionality is now encapsulated within the new profiles.
Optimized AI/ML Storage: This change aims to streamline and potentially optimize the GCSFuse configurations for various AI/ML-specific storage interactions within the A3 Ultra GPU Slurm blueprint.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing several explicit caching and metadata options.

Activity

gemini-code-assist[bot] provided an initial summary of changes for the pull request.
gemini-code-assist[bot] commented on a duplicate implicit_dirs entry in mount_options for a data bucket, suggesting its removal.
gargnitingoogle responded to the implicit_dirs comment, marking the code as "obsolete code."
gemini-code-assist[bot] suggested improvements for a Go installation script, focusing on cleanup, PATH configuration, and readability.
gargnitingoogle responded to the Go installation script comment, marking the code as "obsolete code."
gemini-code-assist[bot] identified a commented-out line in a Terraform module's condition block, recommending its removal for clarity.
gargnitingoogle responded to the Terraform module comment, marking the code as "obsolete code."
gargnitingoogle requested a new summary from gemini-code-assist[bot].

gargnitingoogle · 2026-01-05T04:27:20Z

/gemini summary

gargnitingoogle · 2026-01-05T04:27:25Z

/gemini review

gemini-code-assist · 2026-01-05T04:27:42Z

Summary of Changes

This pull request streamlines the GCSFuse mount configurations within several AI/ML Slurm cluster blueprints. By adopting specific GCSFuse profiles for checkpointing, training data, and model serving, it replaces numerous explicit caching and metadata settings with a more concise and standardized approach. This change enhances the readability and maintainability of the storage configurations across the blueprints.

Highlights

GCSFuse Profile Adoption: Switched GCSFuse mount options in AI/ML Slurm blueprints to use predefined profiles (aiml-checkpointing, aiml-training, aiml-serving) for various GCS bucket mounts.
Simplified Mount Options: Replaced verbose, explicit caching and metadata GCSFuse options (e.g., metadata-cache-*, file-cache-*, enable-streaming-writes) with concise profile-based configurations.
Configuration Standardization: Standardized GCSFuse configurations across multiple machine learning blueprints (a3-ultragpu-8g, a4-highgpu-8g, a4x-highgpu-4g) for improved clarity and maintainability.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing explicit caching and metadata settings.
examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing explicit caching and metadata settings.
examples/machine-learning/a4x-highgpu-4g/a4xhigh-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing explicit caching and metadata settings.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing explicit caching and metadata settings.

Activity

gemini-code-assist[bot] provided an initial summary of changes for the pull request.
gemini-code-assist[bot] commented on a duplicate implicit_dirs entry in mount_options for a data bucket.
gargnitingoogle responded to the implicit_dirs comment, marking the code as "obsolete code."
gemini-code-assist[bot] suggested improvements for a Go installation script.
gargnitingoogle` responded to the Go installation script comment, marking the code as "obsolete code."
gemini-code-assist[bot] identified a commented-out line in a Terraform module's condition block.
gargnitingoogle responded to the Terraform module comment, marking the code as "obsolete code."
gargnitingoogle requested a new summary from gemini-code-assist[bot].
gemini-code-assist[bot] provided an updated summary focusing on GCSFuse profile adoption.
gargnitingoogle requested another summary.
gargnitingoogle requested a review.

gemini-code-assist

Code Review

This pull request updates several Slurm cluster blueprints to use the gcsfuse profile feature for AI/ML workloads. This is a great improvement as it simplifies the configuration, enhances readability, and aligns with best practices for mounting GCS buckets for checkpointing, training, and serving. However, I've found a critical issue in the implementation. The new multi-line mount_options strings are missing the \ line continuation character. This will cause the YAML parser to insert spaces between the options, leading to an invalid fstab entry and mount failures. I've added comments with suggestions to fix this in all affected files.

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml

examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml

examples/machine-learning/a4x-highgpu-4g/a4xhigh-slurm-blueprint.yaml

gargnitingoogle · 2026-01-06T07:25:04Z

/gemini summary

gemini-code-assist · 2026-01-06T07:25:21Z

Summary of Changes

This pull request refactors the GCSFuse mount configurations within several AI/ML Slurm cluster blueprints. By leveraging the GCSFuse profile feature, it replaces numerous fine-grained cache and config parameters with more concise, profile-based settings. This change aims to simplify and optimize the mount options, making the blueprints shorter, easier to maintain, and ensuring GCSFuse automatically applies optimal values for AI/ML workloads.

Highlights

GCSFuse Profile Adoption: Switched GCSFuse mount options in AI/ML Slurm blueprints to use predefined profiles (aiml-checkpointing, aiml-training, aiml-serving) for various GCS bucket mounts.
Simplified Mount Options: Replaced verbose, explicit caching and metadata GCSFuse options (e.g., metadata-cache-*, file-cache-*, enable-streaming-writes) with concise profile-based configurations, leading to shorter and easier-to-maintain mount options.
Standardized Configuration: Applied these changes across multiple machine learning blueprints (A3-ultra, A4x-high, A4-high) for improved clarity and consistency.
Syntax Correction: Corrected implicit-dirs to implicit_dirs in mount options for consistency with GCSFuse parameter naming conventions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs to change implicit-dirs to implicit_dirs and remove enable-streaming-writes.
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing several explicit caching and metadata options.
examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs to change implicit-dirs to implicit_dirs and remove enable-streaming-writes.
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing several explicit caching and metadata options.
examples/machine-learning/a4x-highgpu-4g/a4xhigh-slurm-blueprint.yaml
- Updated GCSFuse mount options for /gcs to change implicit-dirs to implicit_dirs and remove enable-streaming-writes.
- Updated GCSFuse mount options for /gcs-checkpoints to use profile=aiml-checkpointing, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-training-data to use profile=aiml-training, removing several explicit caching and metadata options.
- Updated GCSFuse mount options for /gcs-model-serving to use profile=aiml-serving, removing several explicit caching and metadata options.

Activity

gemini-code-assist[bot] provided an initial summary of changes for the pull request.
gemini-code-assist[bot] commented on a duplicate implicit_dirs entry, suggested improvements for a Go installation script, and identified a commented-out line in a Terraform module.
gargnitingoogle responded to these comments, marking the code as 'obsolete code'.
gargnitingoogle requested multiple summaries from gemini-code-assist[bot].
gemini-code-assist[bot] provided updated summaries focusing on GCSFuse profile adoption.
gargnitingoogle requested a review.
gemini-code-assist[bot] raised critical comments regarding the multi-line string syntax for mount_options in YAML, suggesting potential issues with newlines being folded into spaces.
gargnitingoogle disputed these critical comments, stating that backslashes were present for line continuation and the comments were 'bad'.

gargnitingoogle · 2026-01-20T11:16:16Z

/gemini review

gemini-code-assist

Code Review

This pull request effectively simplifies the gcsfuse mount options in the Slurm cluster blueprints by leveraging the profile feature for AIML workloads. The changes are applied consistently across the a3-ultragpu-8g, a4-highgpu-8g, and a4x-highgpu-4g blueprints, which improves maintainability. My review focuses on ensuring complete consistency in the naming convention for mount options. I've suggested a minor change to use underscores for all gcsfuse options to enhance readability.

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml

examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml

examples/machine-learning/a4x-highgpu-4g/a4xhigh-slurm-blueprint.yaml

examples/machine-learning/a3-ultragpu-8g/a3ultra-slurm-blueprint.yaml

examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml

… blueprint

…blueprint

… blueprint

…4x, a4 slurm blueprints

bytetwin · 2026-01-28T04:38:30Z

/gcbrun

gemini-code-assist bot reviewed Dec 30, 2025

View reviewed changes

examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml Outdated Show resolved Hide resolved

examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml Outdated Show resolved Hide resolved

modules/file-system/cloud-storage-bucket/main.tf Show resolved Hide resolved

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch 2 times, most recently from 8d520a9 to bdfd10d Compare January 2, 2026 08:13

gargnitingoogle changed the title ~~Gargnitin/blueprints/v1~~ Switch to using profile in aiml gcs-bucket mounts in slurm cluster blueprints Jan 5, 2026

gargnitingoogle changed the title ~~Switch to using profile in aiml gcs-bucket mounts in slurm cluster blueprints~~ Switch to using gcsfuse profile in aiml gcs-bucket mounts in slurm cluster blueprints Jan 5, 2026

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch from bdfd10d to fc76677 Compare January 5, 2026 03:58

gargnitingoogle changed the title ~~Switch to using gcsfuse profile in aiml gcs-bucket mounts in slurm cluster blueprints~~ Switch to using gcsfuse profile feature in aiml gcs-bucket mounts in slurm cluster blueprints Jan 5, 2026

gemini-code-assist bot reviewed Jan 5, 2026

View reviewed changes

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch from 2a8da14 to 525af5e Compare January 6, 2026 07:08

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch from 525af5e to 433915b Compare January 20, 2026 11:09

gargnitingoogle marked this pull request as ready for review January 20, 2026 11:12

gargnitingoogle requested review from a team and samskillman as code owners January 20, 2026 11:12

gemini-code-assist bot reviewed Jan 20, 2026

View reviewed changes

cboneti added the release-key-new-features Added to release notes under the "Key New Features" heading. label Jan 21, 2026

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch from 433915b to 1c37225 Compare January 22, 2026 10:40

bytetwin reviewed Jan 27, 2026

View reviewed changes

gargnitingoogle added 4 commits January 27, 2026 03:49

use gcsfuse profiles in the base blueprint for a3-ultra

02f8c2a

replace hyphens with underscores in gcsfuse mount_options in a3-ultra…

72dd082

… blueprint

use gcsfuse profiles in the slurm cluster blueprint for a4-high

b6cb779

replace hyphens with underscores in gcsfuse mount_options in a4-high …

2f993bf

…blueprint

gargnitingoogle added 3 commits January 27, 2026 03:49

use gcsfuse profiles in the slurm cluster blueprint for a4x-high

91cb8f4

replace hyphens with underscores in gcsfuse mount_options in a4x-high…

4d29a19

… blueprint

switch hyphens to underscores in gcsfuse mount_options in a3-ultra, a…

941997d

…4x, a4 slurm blueprints

gargnitingoogle force-pushed the gargnitin/blueprints/v1 branch from 1c37225 to 941997d Compare January 27, 2026 03:49

address review comments

8f5b45a

gargnitingoogle requested a review from bytetwin January 27, 2026 04:59

bytetwin approved these changes Jan 28, 2026

View reviewed changes

cboneti mentioned this pull request Jan 28, 2026

feat: Add sbatch/salloc support to gcsfuse_spank plugin GoogleCloudPlatform/slurm-gcp#306

Merged

cboneti approved these changes Jan 29, 2026

View reviewed changes

cboneti enabled auto-merge January 29, 2026 11:53

cboneti merged commit 6e371ec into GoogleCloudPlatform:develop Jan 29, 2026
15 of 83 checks passed

gargnitingoogle deleted the gargnitin/blueprints/v1 branch January 29, 2026 18:39

Conversation

gargnitingoogle commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Submission Checklist

Uh oh!

gemini-code-assist bot commented Dec 30, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gargnitingoogle commented Jan 2, 2026

Uh oh!

gemini-code-assist bot commented Jan 2, 2026

Summary of Changes

Highlights

Uh oh!

gargnitingoogle commented Jan 5, 2026

Uh oh!

gargnitingoogle commented Jan 5, 2026

Uh oh!

gemini-code-assist bot commented Jan 5, 2026

Summary of Changes

Highlights

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gargnitingoogle commented Jan 6, 2026

Uh oh!

gemini-code-assist bot commented Jan 6, 2026

Summary of Changes

Highlights

Uh oh!

gargnitingoogle commented Jan 20, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bytetwin commented Jan 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

gargnitingoogle commented Dec 30, 2025 •

edited

Loading