Ramp Scales GPU AI Workloads Using AWS ECS Managed Instances

Ramp initially used an "ECS on EC2" pattern for GPU workloads, involving manual provisioning of Auto Scaling groups, launch templates, AMIs, and monitoring scripts. This approach led to significant operational overhead, inconsistent patching, and separate management from their main ECS infrastructure.

Ramp transitioned to Amazon ECS Managed Instances, a fully managed compute option combining EC2 flexibility with Fargate's simplicity. AWS now handles instance configuration, scaling, patching, and maintenance, integrating GPU infrastructure seamlessly with their existing ECS environment.

Bore, Ramp’s internal ML inference platform, uses a mixed compute pattern: CPU-bound services run on Fargate, while GPU-backed services like text similarity and merchant matching run on ECS Managed Instances, targeting NVIDIA L40S GPUs using g6e instances in production.

The Embeddings service hosts multiple vector embedding models, provisioning dedicated capacity providers for different hardware requirements. It uses NVIDIA A10G (g5.xlarge) for default models and NVIDIA L40S (g6e instances) for larger models like Qwen3-Embedding and Gemma-300M.

Ramp organizes its ECS infrastructure with a base Terraform workspace per environment, provisioning cluster-level resources and capacity providers. GPU services declare a capacity provider strategy pointing to specific providers, while non-GPU services continue to use Fargate within the same clusters.