AWS Blog Details Centralized CloudTrail Monitoring on OpenSearch
This AWS Big Data Blog post details how organizations can build a centralized CloudTrail monitoring solution using Amazon OpenSearch Service to manage logs from over 100 accounts. The solution, managed by Terraform, addresses challenges in security monitoring, compliance reporting, and role-based access for multiple teams. It highlights an architecture that scales to handle 200 GB of CloudTrail logs daily, provides automated threat detection, and ensures isolated access for four distinct teams.
- →Problem Statement: Centralized Security Monitoring at Scale
- →Solution Architecture Overview
- →Infrastructure as Code with Terraform
- →Workload Profile and Domain Sizing
- →Serverless Ingestion and Index Management
Notes (6) ›
- Problem Statement: Centralized Security Monitoring at Scale
Organizations with numerous AWS accounts struggle with centralized security monitoring, requiring manual log searching and custom Lambda functions for tasks like index lifecycle management and compliance reporting, which is time-consuming and error-prone.
- Solution Architecture Overview
The proposed architecture aggregates CloudTrail logs from 100+ accounts into a central S3 bucket, using SQS notifications to trigger an auto-scaling Amazon OpenSearch Ingestion pipeline. The solution includes an Amazon OpenSearch Service domain with fine-grained access control and tenant isolation for four teams.
- Infrastructure as Code with Terraform
Terraform is used to manage the full stack, including domain provisioning, access control, lifecycle policies, index templates, alerting monitors, and dashboards, ensuring consistency and programmatic governance across environments.
- Workload Profile and Domain Sizing
A workload profile defining index throughput (200 GB/day), search latency (<100 ms), and retention (30 days) informed the decision to use storage-optimized OR1 instances for the OpenSearch domain, prioritizing indexing throughput. The post notes OR2 instances are recommended for new deployments due to higher throughput.
- Serverless Ingestion and Index Management
Amazon OpenSearch Ingestion provides serverless, auto-scaling ingestion from S3, scaling between 2 and 10 OCUs based on queue depth. Index templates define explicit mappings, and a declarative Index State Management (ISM) policy automates rollover based on index age or size, reducing custom code by ~60%.
- Multi-Team Access Control and Tenant Isolation
Role-based access is managed declaratively in Terraform, mapping IAM roles to OpenSearch roles. Each team gets an isolated tenant in OpenSearch Dashboards, preventing interference and maintaining infrastructure-as-code governance for visualizations and configurations.
https://aws.amazon.com/blogs/big-data/centralized-cloudtrail-monitoring-across-100-aws-accounts/
Related releases
- Terraform AWS Provider v6.60.0 Adds New Resources and Fixes Bugs Terraform AWS Provider Releases ·
- Amazon S3 enhances access denied errors with specific policy ARNs AWS What's New ·
- OpenAI Daybreak Red and Blue Cyber Defense Models Now on Amazon Bedrock AWS What's New ·
- AppFolio Transforms Data Streaming with Amazon MSK Express Brokers AWS Big Data Blog ·
- AWS IAM Role Manager Automates Service Role Setup AWS What's New ·
- Amazon Quick Integrates Microsoft Purview for Data Loss Prevention AWS What's New ·