AWS Big Data Blog details talabat's hybrid multi-cloud lakehouse for real-time analytics
The AWS Big Data Blog showcases how talabat, a prominent app in the MENA region, developed a hybrid multi-cloud lakehouse architecture. This solution allows talabat to maintain a single Apache Iceberg copy of streaming data on AWS while enabling governed, near real-time analytics from Google Cloud Platform's BigQuery. The architecture overcomes challenges of cross-cloud and cross-region data movement, significantly reducing latency, data duplication, and egress costs by allowing BigQuery to read data in place. This approach is critical for optimizing real-time decisions in pricing, dispatch, routing, and order security for talabat's extensive customer base.
- →talabat's Multi-Cloud Data Challenge
- →Limitations of Traditional Cross-Cloud Data Movement
- →Adopting a Hybrid Data Lakehouse Approach
- →Strategic Justification for a Hybrid Data Footprint
- →Lessons from Early Attempts at Cross-Cloud Writes
Notes (6) ›
- talabat's Multi-Cloud Data Challenge
Talabat, a leading MENA app, operates its transactional systems on AWS and analytics on Google Cloud, creating a strategic challenge for efficient data flow across these distinct environments.
- Limitations of Traditional Cross-Cloud Data Movement
Historically, talabat's data engineering involved physical data movement via ETL from AWS to GCP, resulting in multiple hops, data duplication, compounding latency, and significant cross-region and cross-cloud egress costs.
- Adopting a Hybrid Data Lakehouse Approach
To overcome previous limitations, talabat redesigned its architecture with a premise to keep a single copy of streaming data on AWS while enabling Google BigQuery to read it in place, forming a hybrid, multi-cloud lakehouse.
- Strategic Justification for a Hybrid Data Footprint
Talabat determined that its event backbone on AWS and analytics estate on Google Cloud each played to genuine strengths, making consolidation impractical and necessitating engineering the seam between the two clouds instead of removing it.
- Lessons from Early Attempts at Cross-Cloud Writes
An initial attempt to write raw data directly from AWS to BigQuery-managed Iceberg tables on Google Cloud Storage failed due to introducing cross-cloud dependencies, latency on the ingestion path, and write-API failures impacting ingestion.
- Implementing Federated Apache Iceberg for Cross-Cloud Reads
Talabat adopted a lakehouse federated Apache Iceberg REST catalog, synchronizing AWS Glue Data Catalog metadata with IAM authentication, allowing BigQuery to query AWS-resident data in Amazon S3 directly without data movement.
https://aws.amazon.com/blogs/big-data/fresher-insights-faster-decisions-talabats-near-real-time-analytics-across-aws-and-google-cloud/
Related releases
- Amazon Connect Customer adds information extraction for agent voice and chat AWS What's New ·
- Amazon Redshift now offers long-term system table retention via S3 Tables AWS What's New ·
- CloudFront OAC now natively supports S3 Multi-Region Access Points as origins AWS What's New ·
- Amazon Redshift adds long-term system table retention with S3 Tables AWS Big Data Blog ·
- SageMaker AI Studio Adds Generative AI Inference Recommendations for Optimized Deployments AWS What's New ·
- Amazon Bedrock now supports SpaceXAI Grok 4.6 with Cross-Region Inference AWS What's New ·