AWS re:Invent 2024 - Accelerate production for gen AI using Amazon SageMaker MLOps & FMOps (AIM354)

Introduction

In this session aimed at data scientists and machine learning engineers, AWS focuses on helping professionals deploy generative AI use cases to production quickly and reliably. The discussion began by highlighting three key trends emerging with generative AI:

1. Rapid Growth in Spending: Customer spending on generative AI surged from $ 7 million to $ 18 million annually in less than a year—indicative of the technology's increased adoption.
2. Diverse Model Providers: Companies are increasingly adopting a multi-provider strategy for building generative AI applications.
3. Rising Adoption of Open-Source Foundation Models: With the availability of high-quality open-source foundation models, businesses are opting to customize these pre-trained models rather than develop their own from scratch.

The foundation of operationalizing any AI/ML workload is governance, which spans across AWS services, data, models, and applications, requiring constant oversight from development through deployment, monitoring, and auditing. This governance strategy ensures that companies can manage data access and maintain the quality and accuracy of training datasets while enforcing privacy.

Overview of MLOps

Amit Modi, Senior Manager of Product Management for SageMaker, acknowledged the significance of MLOps. He highlighted MLOps as a framework built on governance that includes practices for scaling, repeatability, and reliability across workflows. Within SageMaker, key MLOps capabilities include:

• Automated experiment tracking
• Model management at scale
• Building pipelines for model training and fine-tuning
• Deployment and monitoring of model performance
• Documentation of use cases for compliance and governance

The discussion transitioned into Foundation Model Operations (FM Ops) which builds upon MLOps by involving unique components relevant to foundation models, such as:

• Selecting and evaluating foundation models
• Prompt modifications to manage model behavior
• Implementation of safeguards against harmful content

Generative Operations (Gen Ops) further extends FM Ops by introducing unique challenges and techniques related to the end-to-end implementation of generative AI solutions, such as building agents, augmenting models with secondary data sources, and tracing application behavior in production.

Identified Challenges in FM Ops and Gen Ops

Following this, common challenges arising from conversations with customers were discussed, offering actionable strategies to navigate these hurdles. Notably, the need for high-quality fine-tuning data was emphasized, as it often relies on human annotators. By utilizing tools like SageMaker Ground Truth, businesses can manage feedback in a user-friendly interface integrated into development pipelines.

Experiment and Model Management

Reliable experiments and model management are critical, and managed MLflow on SageMaker helps alleviate customer concerns about managing infrastructure while allowing for seamless upgrades and integrations. Key capabilities provided by SageMaker include:

• Easy launch and configuration of MLflow tracking servers.
• Automated syncing of models between MLflow and SageMaker Model Registry, allowing for easy deployment.

Prompt Management

With the rise of prompt management, users want seamless interaction with foundation models. Utilizing prompt templates and associated evaluation results fosters collaboration among different teams, allowing for consistent practices that enhance efficiency across experiments.

Building Repeatable Workloads

SageMaker Pipelines enables the creation of automated and repeatable workflows, integrating training, processes, and model registration. Each component can be executed seamlessly, allowing for scalability and reduced redundancy.

Evaluation and Monitoring

Continuous evaluation and monitoring of generative applications are paramount, with the emphasis on using appropriate metrics for thorough assessment. Leveraging SageMaker Clarify and MLflow offers customers the tools to evaluate model performance effectively.

Governance and Safeguards

Governance and risk compliance in deploying models provide a systematic way to track artifacts and ensure smooth operational transitions. Various AWS services support logging and monitoring to enforce governance.

Cost-Effective Deployments

Amazon emphasizes the importance of having cost-efficient models, providing optimized compute options across different layers of machine learning operations, ensuring businesses can deploy models efficiently.

Case Study: Rocket Mortgage

The presentation featured insights from Diane, Director of Engineering at Rocket Mortgage, showcasing their journey in adopting advanced AI strategies. Despite initial struggles with a cumbersome deployment process involving several machine learning engineers, Rocket Mortgage successfully transitioned to using SageMaker and MLflow, significantly reducing their development time by 40-60%. As a result, they scaled up to over 200 proprietary AI models in production, fueling efficiency and improved customer engagement.

Closing Remarks

Shelby wrapped up the session with a summary of key takeaways and resources available to attendees, paving the way for further explorations in MLOps, FM Ops, and Gen Ops.

---

Keywords

• Generative AI
• MLOps
• FM Ops
• Gen Ops
• SageMaker
• MLflow
• Foundation Models
• Experiment Tracking
• Prompt Management
• Governance

FAQ

Q1: What is the main focus of the AWS re:Invent 2024 session AIM354?
A: The session aims to help data scientists and machine learning engineers deploy generative AI use cases quickly and reliably using Amazon SageMaker's MLOps and FM Ops capabilities.

Q2: What are the key trends in generative AI highlighted in the session?
A: The key trends include rapid growth in customer spending, reliance on multiple model providers, and the use of high-quality open-source foundation models.

Q3: What challenges do companies face in implementing FM Ops and Gen Ops?
A: Common challenges include obtaining high-quality fine-tuning data, managing model experiments, and implementing governance and risk management practices.

Q4: What tools does SageMaker offer to support experiment tracking?
A: SageMaker provides managed MLflow capabilities for reliable experiment tracking and model management.

Q5: How has Rocket Mortgage benefited from adopting SageMaker and MLflow?
A: Rocket Mortgage reduced its development time by 40-60% and scaled to operationalize over 200 proprietary AI models, enhancing efficiency and customer interactions.