aws AWS Big Data Blog ·

Python and Bash Operators Now Available in Amazon MWAA Serverless

blogdataawsgaengineer
feature

Amazon Managed Workflows for Apache Airflow (MWAA) Serverless now supports `PythonOperator` and `BashOperator`, allowing users to run custom Python functions and shell scripts directly within the serverless runtime. This enhancement eliminates the need to encapsulate custom logic in external compute services like AWS Lambda or Amazon ECS tasks, reducing complexity, cost, and latency for data orchestration pipelines. Users can package custom code into S3-uploaded bundles, which are snapshotted at workflow creation for consistent execution. While tasks have limited default network access, VPC configuration can enable broader connectivity.

Features (1)
  • Native Python and Bash Operator Support in MWAA Serverless

    Amazon MWAA Serverless now allows users to run custom Python functions and shell scripts directly within their workflows using the `PythonOperator` and `BashOperator`. This new capability streamlines the execution of custom code, removing the prior requirement to integrate external AWS compute services for such logic. Code bundles are uploaded to Amazon S3 and snapshotted for reliable, isolated execution, significantly reducing pipeline complexity and operational overhead.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/pythonoperator-and-bashoperator-now-available-on-amazon-managed-workflows-for-apache-airflow-amazon-mwaa-serverless/

Related releases