AWS Post Details Automated Evaluation Pipeline for Connect AI Agents
This AWS blog post details how to build an automated evaluation pipeline for Amazon Connect Customer AI Agents using DeepEval and Amazon Bedrock. This pipeline addresses Model Risk Management (MRM) requirements, particularly for financial institutions adhering to SR 11-7 guidance, by enabling quantitative, auditable testing of AI agent performance. The approach uses an LLM-as-judge hosted on Amazon Bedrock for nuanced evaluation, ensuring test inference remains within the AWS account for security and compliance. The walkthrough covers deploying a two-layer pipeline to test both tool execution and conversational flow, generating reports suitable for MRM documentation.
- →Leveraging DeepEval and Amazon Bedrock for LLM Evaluation
- →Two-Layer Evaluation Pipeline
- →Mapping Metrics to MRM Validation Points
- →Automated Evaluation for Model Risk Management (MRM)
- →Challenges of Manual AI Agent Testing
Features (3) ›
- Leveraging DeepEval and Amazon Bedrock for LLM Evaluation
The solution utilizes DeepEval, an open-source LLM evaluation framework, and Amazon Bedrock to build an automated pipeline. This pipeline generates quantitative artifacts like semantic similarity scores, routing accuracy, and conversation outcome metrics, using an LLM-as-judge hosted within the AWS account.
- Two-Layer Evaluation Pipeline
A two-layer evaluation pipeline is presented: a Gateway layer testing deterministic tool execution via AgentCore Gateway, and an Amazon Connect Customer layer testing the full end-to-end conversational stack, including intent classification and LLM response generation.
- Mapping Metrics to MRM Validation Points
The DeepEval harness maps metrics like GEval Correctness and AnswerRelevancy to key MRM validation points, specifically User Input Interpretation and Conversation Outcome. GEval uses chain-of-thought prompting to assess the semantic equivalence and correctness of the agent's response against expected outcomes.
Notes (3) ›
- Automated Evaluation for Model Risk Management (MRM)
The post demonstrates how to create an automated evaluation pipeline for Amazon Connect Customer AI Agents, crucial for meeting Model Risk Management (MRM) requirements, especially those outlined in the U.S. Federal Reserve’s SR 11-7 guidance. This addresses the need for independent validation of AI models influencing customer outcomes before production deployment.
- Challenges of Manual AI Agent Testing
Manual testing of AI agents is insufficient for MRM at scale, as it lacks repeatable execution records, quantitative scoring, and coverage evidence for edge cases. Human testers introduce variability, increase costs, and slow down the release of new features.
- Secure and Compliant LLM-as-Judge Deployment
Running the LLM judge on Amazon Bedrock ensures that all evaluation inference remains within the customer's AWS account, adhering to their security controls, VPC configurations, and data residency policies. This is particularly important when dealing with sensitive banking-domain data.
https://aws.amazon.com/blogs/contact-center/evaluating-amazon-connect-customer-ai-agents-with-deepeval-for-mrm/
Related releases
- OpenAI Daybreak Red and Blue Cyber Defense Models Now on Amazon Bedrock AWS What's New ·
- Amazon Bedrock expands IAM principal cost allocation for mantle endpoint AWS What's New ·
- AWS AI agent optimizes Kinesis Data Streams costs AWS Big Data Blog ·
- Deploying .NET AI Agents with Amazon Bedrock AgentCore AWS Developer Tools Blog ·
- AWS Weekly Roundup: Web Search, DynamoDB Vector Search, Lambda Bandwidth, and AI Agent Tools AWS News Blog ·
- Amazon Bedrock AgentCore gains memory, policy, and managed harness in GovCloud AWS What's New ·