AWS Tools for PowerShell v5 Adds Multipart Download for S3
AWS Tools for PowerShell v5 now supports multipart downloads from Amazon S3, significantly improving performance for large object transfers. This opt-in feature uses the AWS SDK for .NET's S3 Transfer Manager to enable faster downloads via parallel connections and concurrent streams, reducing the complexity of manual stream management. The enhancement applies to both single object and directory downloads using `Read-S3Object` and `Copy-S3Object` cmdlets, benefiting engineers and architects who regularly handle large datasets.
- →Multipart download support for `Read-S3Object` and `Copy-S3Object`
- →Download strategies: Part number and byte-range fetches
- →Customizable parallelism and download settings
- →S3-to-local copy operations gain multipart support
- →Migration path for existing scripts
Features (1) ›
- Multipart download support for `Read-S3Object` and `Copy-S3Object`
The `Read-S3Object` and `Copy-S3Object` cmdlets in AWS Tools for PowerShell v5 have been updated to support multipart downloads. This feature improves download performance by enabling parallel downloads of large objects from Amazon S3, utilizing concurrent connections and managed retry logic. An opt-in `-UseMultipartDownload` switch parameter is available to enable this functionality.
Enhancements (3) ›
- Download strategies: Part number and byte-range fetches
Two download strategies are now supported: part number fetches, which work with objects uploaded using standard multipart upload part sizes, and byte-range fetches, which can download any S3 object regardless of its upload method. Byte-range fetches facilitate greater parallelization, especially for objects with large parts, though smaller range sizes increase the number of S3 requests and associated costs.
- Customizable parallelism and download settings
Users can customize multipart download configurations, including part size for byte-range fetches and the number of concurrent service requests (defaulting to 10). For directory downloads, a `-DownloadFilesConcurrently` parameter enables file-level parallelism, further boosting throughput for directories containing multiple large files.
- S3-to-local copy operations gain multipart support
The `Copy-S3Object` cmdlet now supports the same multipart download parameters when copying objects from Amazon S3 to a local file. These parameters are specific to S3-to-local operations and do not apply to S3-to-S3 copy operations.
Maintenance (1) ›
- Migration path for existing scripts
Existing download scripts will continue to function but will return legacy response objects. Migrating to the new multipart support provides access to enhanced S3 response metadata and performance benefits. The update requires installing version 5.0.208 or later of the AWS Tools for PowerShell modules.
https://aws.amazon.com/blogs/developer/introducing-multipart-download-support-for-aws-tools-for-powershell-v5/
Related releases
- Amazon S3 enhances access denied errors with specific policy ARNs AWS What's New ·
- AWS Clean Rooms now supports exporting privacy-enhanced analysis logs for SQL queries AWS What's New ·
- AWS Security Blog Details How to Secure S3 Buckets AWS Security Blog ·
- AWS Glue integrates with SageMaker Unified Studio for one-click data access AWS What's New ·
- AWS Backup for S3 adds direct read-only access to backup data AWS What's New ·
- Amazon MSK adds authorizer log delivery for provisioned clusters AWS What's New ·