ReleaseBytes
aws AWS Big Data Blog ·

Moeve Standardizes dbt Data Lake Workflows with Amazon Athena

blogdataawsarchitectaws-dynamodb
announcement

Moeve implemented a centralized, serverless solution for dbt runs on Amazon Athena to address fragmented data processing and inconsistent compute across its data lakes. This standardization improved cost visibility and operational efficiency, reducing new dbt project onboarding from days to just 15 minutes. The architecture leverages Athena's serverless capabilities, Apache Iceberg for optimized incremental processing, and integrates with AWS Step Functions, Fargate, DynamoDB, and EventBridge.

  • →Moeve standardizes dbt runs across data lakes with Amazon Athena
  • →Serverless architecture for cost-effective, idempotent transformations
Notes (2) ›
  • Moeve standardizes dbt runs across data lakes with Amazon Athena

    Moeve developed a centralized, serverless dbt launcher using Amazon Athena to overcome fragmented data processing, inconsistent compute, and limited cost visibility across its data lakes. This standardized approach significantly reduced onboarding time for new dbt projects from days to approximately 15 minutes.

  • Serverless architecture for cost-effective, idempotent transformations

    The solution leverages Amazon Athena as the serverless processing engine, chosen for its operational simplicity and pay-per-query model. The architecture uses AWS Step Functions for run lifecycle control, AWS Fargate for ephemeral dbt container execution, Amazon DynamoDB for project configuration, and Amazon EventBridge to publish run results. It also features optimized incremental processing for Apache Iceberg tables, building dynamic predicates to reduce scanned data and achieve idempotent operations.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/how-moeve-standardized-dbt-runs-across-data-lakes-with-amazon-athena/

Related releases