Google Cloud Managed Lustre
High performance parallel file system
Accelerate HPC and AI training and serving with Google's highest performance, POSIX-compliant, parallel file system.
Features
Performance and scale for AI/ML workloads
Training large deep learning models requires massive datasets. Managed Lustre, based on DDN EXAScaler, distributes data access, reducing training times, enabling faster insights, better accuracy, and handling of complex AI projects. Its scalability is designed to performance keeps pace with growing data, preventing storage bottlenecks. Watch Omdia overview on Managed Lustre here.
Accelerate AI inference with Key-Value cache
Agentic AI continues to drive large context windows and this may pose a storage challenge to give you with a responsive experience as they interact with Large Language Models. Large context windows increase latency sensitivity as the local memory on the accelerators is often exhausted, requiring the model to access external storage.
Maximize GPU saturation and compute ROI
The true cost of AI isn't just storage—it's idle compute. Managed Lustre delivers the high-throughput, low-latency data delivery required to keep your most expensive assets fully saturated. By optimizing data distribution and accelerating rapid checkpointing, you dramatically improve accelerator use, leading to reduced overhead and superior performance per dollar.
Powering innovation across industries
| Industries |
|---|
AI and ML Eliminate data starvation and train foundation models at massive scale. By decoupling compute and storage, Managed Lustre boosts LLM inference throughput, with sub-millisecond external Key-Value caching, keeping your most expensive accelerators fully saturated. Explore AI at Google Cloud. |
Healthcare and life sciences Propel groundbreaking innovation toward new cures. Deliver the extreme IOPS required to accelerate drug discovery, analyze complex genomics sequencing, and power AI-driven medical imaging suites, drastically reducing time-to-insight for researchers and clinicians. Explore healthcare and life sciences at Google Cloud. |
Machine vision, robotics, and autonomous vehicles Accelerate the software-defined vehicle pipeline. Empower innovators to effortlessly ingest petabyte-scale sensor telemetry, while supercharging research and development engineering with zero-latency storage for aerodynamics, safety, and, thermal optimization simulations. Explore automotive at Google Cloud. |
Capital markets Execute workloads that demand sub-millisecond precision. Whether running intricate quantitative risk analyses and real-time market simulations for financial services, accelerating high-resolution VFX rendering and, post-production workflows for global media studios. Explore capital markets and financial services at Google Cloud. |
Media and entertainment Meet production deadlines with zero-compromise storage performance. Give your studios with the ultra-high throughput necessary for seamless high-resolution video editing, real-time VFX rendering, and, accelerated post-production workflows. Explore media and entertainment at Google Cloud. |
AI and ML
Eliminate data starvation and train foundation models at massive scale. By decoupling compute and storage, Managed Lustre boosts LLM inference throughput, with sub-millisecond external Key-Value caching, keeping your most expensive accelerators fully saturated.
Explore AI at Google Cloud.
Healthcare and life sciences
Propel groundbreaking innovation toward new cures. Deliver the extreme IOPS required to accelerate drug discovery, analyze complex genomics sequencing, and power AI-driven medical imaging suites, drastically reducing time-to-insight for researchers and clinicians.
Explore healthcare and life sciences at Google Cloud.
Machine vision, robotics, and autonomous vehicles
Accelerate the software-defined vehicle pipeline. Empower innovators to effortlessly ingest petabyte-scale sensor telemetry, while supercharging research and development engineering with zero-latency storage for aerodynamics, safety, and, thermal optimization simulations.
Explore automotive at Google Cloud.
Capital markets
Execute workloads that demand sub-millisecond precision. Whether running intricate quantitative risk analyses and real-time market simulations for financial services, accelerating high-resolution VFX rendering and, post-production workflows for global media studios.
Explore capital markets and financial services at Google Cloud.
Media and entertainment
Meet production deadlines with zero-compromise storage performance. Give your studios with the ultra-high throughput necessary for seamless high-resolution video editing, real-time VFX rendering, and, accelerated post-production workflows.
Explore media and entertainment at Google Cloud.
Common Uses
Get started
- Create Managed Lustre instance: Deploy your Managed Lustre instance with just a few clicks.
- Ingest and Connect: Easily hydrate your Managed Lustre file system with data directly from Google Cloud Storage.
- Process at Scale: Feed massive datasets into Vertex Training Clusters (VTC) or GKE with ultra-low latency.
- Accelerate: Give high-throughput data directly to next-generation hardware, ensuring GPUs run at maximum use.
Tutorials, quickstarts, & labs
- Create Managed Lustre instance: Deploy your Managed Lustre instance with just a few clicks.
- Ingest and Connect: Easily hydrate your Managed Lustre file system with data directly from Google Cloud Storage.
- Process at Scale: Feed massive datasets into Vertex Training Clusters (VTC) or GKE with ultra-low latency.
- Accelerate: Give high-throughput data directly to next-generation hardware, ensuring GPUs run at maximum use.
Pricing
| Managed Lustre pricing | Pricing for Managed Lustre is primarily based on location and service level. |
|---|---|
| Service level | Pricing |
1,000 MB/TB Best for high-performance workloads want AI/ML training where throughput is critical. | Starting at $0.60 per GB per month |
500 MB/TB Best for high-performance balance: Excellent for demanding AI/ML workloads, complex HPC applications, and data-intensive analytics that require substantial throughput but may benefit from a more balanced price-to-performance ratio. | Starting at $0.34 per GB per month |
250 MB/TB Best for general purpose HPC and throughput-intensive AI: Suitable for a broad range of HPC workloads, AI/ML inference, data preprocessing, and applications that require significantly better performance than traditional NFS, at a cost-effective price point. | Starting at $0.21 per GB per month |
125 MB/TB Best for capacity-focused workloads with parallel access requires: Designed for scenarios where large capacities and parallel file system access are key. Good for less I/O-bound parallel tasks. | Starting at $0.145 per GB per month |
Explore Google Cloud pricing. View all pricing details.
Managed Lustre pricing
Pricing for Managed Lustre is primarily based on location and service level.
1,000 MB/TB
Best for high-performance workloads want AI/ML training where throughput is critical.
Starting at $0.60 per GB per month
500 MB/TB
Best for high-performance balance: Excellent for demanding AI/ML workloads, complex HPC applications, and data-intensive analytics that require substantial throughput but may benefit from a more balanced price-to-performance ratio.
Starting at $0.34 per GB per month
250 MB/TB
Best for general purpose HPC and throughput-intensive AI: Suitable for a broad range of HPC workloads, AI/ML inference, data preprocessing, and applications that require significantly better performance than traditional NFS, at a cost-effective price point.
Starting at $0.21 per GB per month
125 MB/TB
Best for capacity-focused workloads with parallel access requires: Designed for scenarios where large capacities and parallel file system access are key. Good for less I/O-bound parallel tasks.
Starting at $0.145 per GB per month
Explore Google Cloud pricing. View all pricing details.
Business Case
Hear from our Managed Lustre customers
View our latest launches:
“Our ability to help companies identify and block deepfake audio, video, and images is only as good as our models. Managed Lustre is critical to our successful model training with our dynamic datasets. We may fully saturate our GPUs and it's 6x faster than the other storage solutions we evaluated.”
—Zohaib Ahmed, CEO Resemble AI
“Managed Lustre enables us to scale AI model training for AFEELA Intelligent Drive by 3x compared to other Google Cloud solutions.”
—Motoi Kataoka, Senior Manager, AI and Data Analytics Platform, Sony Honda Mobility Inc.
"By integrating Managed Lustre with VTC (vertex training clusters), Salesforce AI Research eliminated the typical onboarding bottlenecks, allowing us to hit the ground running with the inferencing workload. This high-throughput low-latency storage keeps our B200 GPUs fully saturated, driving a substantial performance gain in Large Language Models inference over the H200. For our customers, this translates directly into faster, more responsive AI agents that may handle complex reasoning at a fraction of the previous latency."
—Lavanya Karanam, Principal Software Engineer, Salesforce
Scaling GKE workloads with Managed Lustre
A guide on using the Managed Lustre CSI driver with Google Kubernetes Engine (GKE) to seamlessly provision high-performance storage for containerized AI, ML, and HPC workloads. Read Blog →
Accelerating AI and HPC with Managed Lustre
Overview of how Managed Lustre simplifies the deployment of parallel file systems for high-performance computing workloads. Read Blog →
External Key-Value cache with Managed Lustre
A deep dive into using Lustre to offload key-value caches for large language model (LLM) inference, reducing memory overhead on TPUs/GPUs. Read Blog →
