AWS EMR and Spark: Reduce KMS Decrypt API Costs

Consolidating numerous small files into fewer, larger files reduces the total number of AWS KMS API invocations. This technique is effective for read-heavy workloads involving many small files stored on S3, directly lowering operational costs by minimizing decrypt operations per file.

Selecting appropriate file formats like columnar Parquet/ORC and using compression (e.g., ZSTD, Snappy) minimizes the amount of data read from S3 and reduces the number of AWS KMS decrypt operations. Apache Iceberg is highlighted as a modern table format that simplifies data management and optimizes file layouts for large-scale analytic datasets.

Utilizing AWS Glue Data Catalog partition indexes enables server-side filtering of partitions, reducing the number of S3 objects scanned and consequently decreasing AWS KMS API calls. This approach is beneficial for partitioned tables where AWS Glue returns only matching partitions, unlike the traditional method which scans all partitions and objects.

Organizations processing large amounts of encrypted Amazon S3 data on Amazon EMR with Apache Spark face rising costs due to frequent AWS KMS decrypt API calls. Each access to an encrypted S3 object triggers a decrypt call, which accumulates significantly at scale, particularly for workloads requiring per-object auditability that cannot use S3 Bucket Keys.

While the post details techniques for further cost reduction, S3 Bucket Keys are identified as the most effective method to reduce AWS KMS decrypt API calls. The presented optimizations can be used alongside S3 Bucket Keys or independently when per-object auditability requirements prevent their use.