Amazon is aiming to improve AI model evaluation and promote greater human involvement in the process. During the AWS re: Invent conference, AWS Vice President Swami Sivasubramanian introduced Model Evaluation on Bedrock, a preview feature for models within Amazon Bedrock's repository. This initiative addresses the need for transparent model testing to prevent developers from unintentionally selecting inaccurate or oversized models for their projects.

‍

Model Evaluation comprises two components: automated evaluation and human evaluation. In the automated version, developers can assess model performance metrics, such as robustness and accuracy, for various tasks. Bedrock includes third-party AI models like Meta’s Llama 2 and Stability AI’s Stable Diffusion. Additionally, users can bring their own data into the benchmarking platform, allowing them to better understand model behavior and generate reports. If humans are involved, customers can collaborate with an AWS human evaluation team or specify their own criteria, benefiting from customized pricing and timelines.

‍

AWS aims to help customers select the most suitable models, detect responsible AI standards, and identify nuanced metrics like empathy and friendliness. Although not mandatory, benchmarking is valuable for developers exploring model options. AWS will charge only for model inference during the evaluation phase, emphasizing the goal of offering companies a way to gauge a model's impact on their projects rather than broadly evaluating models across industries.