ReleaseBytes
gcp Google Cloud Blog ·

Google Cloud Filestore enhanced with Colossus distributed storage

blogaigcpgaengineergcp-gke
feature

Google Cloud's Filestore now utilizes Colossus, Google's foundational distributed storage system, for its backend. This enhancement provides greater flexibility and scalability, allowing independent provisioning of IOPS and capacity, which is particularly beneficial for demanding AI and containerized workloads on GKE. The upgrade aims to improve operational agility, cost efficiency, and enable new use cases like shared storage for AI agent swarms.

  • →Decoupled storage capacity and performance provisioning
  • →Enhanced GKE integration with Filestore CSI driver and multishares
  • →Shared storage for AI agent swarms with NFS file locking
  • →Enterprise-grade security features integrated with Filestore
  • →Filestore backend upgraded to Google's Colossus distributed storage
Features (4) ›
  • Decoupled storage capacity and performance provisioning

    Filestore now allows independent provisioning of IOPS and storage capacity. This enables users to precisely match performance demands without over-provisioning capacity, supporting a wide range of workloads from development environments to large datasets.

  • Enhanced GKE integration with Filestore CSI driver and multishares

    Filestore provides persistent, high-performance storage for GKE workloads via its CSI driver. Filestore multishares for GKE allows segmentation of instances into smaller shares (starting at 10 GiB), enabling AI teams to scale GKE clusters efficiently and optimize resource utilization.

  • Shared storage for AI agent swarms with NFS file locking

    Filestore now supports high-concurrency workspaces for AI agent swarms, providing consistent file system access via NFS. It ensures data consistency with NFS file locking, enabling agents to maintain a unified view of their environment for complex collaboration.

  • Enterprise-grade security features integrated with Filestore

    Filestore integrates with Google Cloud IAM, NFS User IDs and Group IDs (UIDs/GIDs), and IP Access Control Lists (ACLs) to provide security at scale for enterprise use cases.

Enhancements (2) ›
  • Filestore backend upgraded to Google's Colossus distributed storage

    Filestore now uses Colossus, Google's distributed storage system, for its cloud-native backend. This change leverages the same infrastructure as services like YouTube and Gemini, enabling superior scalability and operational efficiency compared to older architectures.

  • Improved operational agility with independent IOPS scaling and faster recovery

    Users can now independently scale IOPS via Custom Performance settings for real-time cost optimization. The distributed storage layer also offers faster failure recovery and zero-downtime capacity changes compared to monolithic architectures.

Read the original announcement →

https://cloud.google.com/blog/products/storage-data-transfer/filestore-file-service-runs-on-colossus/

Related releases