How Autodesk Migrated 2.3 Billion Documents to Amazon OpenSearch Service

Autodesk's Forma data management component, which supported search across 2.3 billion documents, ran on a single Elasticsearch 7.1.1 domain with one index. This setup led to degraded query performance, vertical scaling limits on Amazon EC2 instance sizes, and horizontal scaling issues due to hot nodes as data grew.

Autodesk re-architected their search service into four Amazon OpenSearch Service domains (OpenSearch 2.19), each running 24 m7i.4xlarge.search nodes and hosting six indices. A routing layer uses a DynamoDB table with 4.3 million records to map projects to specific domains, ensuring queries hit only relevant shards.

The multi-domain architecture offers horizontal scalability by allowing the addition of more domains and indices, efficient routing with queries hitting only specific shards, and a reduced blast radius where only 25% of traffic is affected if a domain fails. It also enables independent scaling of each domain and provides a larger aggregate search-thread pool.

The migration involved categorizing 4.3 million projects by document count and projected growth using historical data. To ensure even data distribution, a stratified bin-packing algorithm was developed, using an interleaved, round-robin approach for different project size categories to avoid imbalance across the 24 new indices.