Posts in category: Amazon AWS

Today, we are excited to announce support for Code Editor, a new integrated development environment (IDE) option in Amazon SageMaker Studio. Code Editor is based on Code-OSS, Visual Studio Code Open Source, and provides access to the familiar environment and tools of the popular IDE that machine learning (ML) developers know and love, fully integrated with the broader SageMaker Studio feature set. Code Editor enables you to choose from thousands of VS Code compatible extensions available in the Open-VSX extension gallery to further enhance your teams’ development experience. You can also maximize your team’s productivity by using seamless integration to AWS services through the AWS Toolkit for Visual Studio Code, including the AWS AI-powered coding companion, Amazon CodeWhisperer.

As with all IDE applications in SageMaker Studio, ML developers and engineers can select the underlying compute on demand, and swap it based on their needs without losing data. Additionally, your teams can manage their codebase version control and collaborate across teams through native GitHub integration and reduce time-to-coding by using the most popular ML frameworks right out of the box with the pre-configured Amazon SageMaker Distribution container image.

Getting started with Code Editor on Amazon SageMaker Studio

Your IT administrator can setup a new SageMaker Studio domain or migrate an existing one to the new SageMaker Studio experience, which includes Code Editor. See Onboard to Amazon SageMaker Domain using Quick setup for more details. You can then launch Code Editor with a simple click in your Amazon SageMaker Studio environment.

1. After the domain is setup, launch SageMaker Studio’s new experience from the console or the pre-signed URL your administrator provided. You can find the Code Editor IDE in both the Applications section in the left-side panel and the Overview section, as shown in the following screenshot:
   
2. On the Code Editor details page, choose Create Code Editor Space. Then enter a name for your space and choose Create space:
   
3. On the Code Editor Space details page, choose your underlying configuration, including:
   1. The underlying Amazon Elastic Compute Cloud (Amazon EC2) instance type.
   2. An Amazon Elastic Block Storage (Amazon EBS) volume size (this can range from 5GB to 16TB).
   3. The container image to use (you will have a SageMaker Distribution image for both CPU and GPU available at launch).
   4. A lifecycle configuration script to run in case you want customize your environment at app creation.
   5. A shared Amazon Elastic File System (Amazon EFS) to mount in your Code Editor space (this needs to be configured by your administrator when provisioning your domain).
4. After providing your space configuration details, choose Run space to provision your space resources.
   

If you have chosen a fast-launch instance with the default SageMaker Distribution as image, your Code Editor space will be available in less than a minute. If you have added lifecycle configurations to the space, then it might take additional time to install dependencies from that script.

After your resources are provisioned, the space details page will show an Open Code Editor button.

5. Choose Open CodeEditor to launch the IDE.
   

The code editor IDE will launch in a new browser tab.

Code Editor features

Code Editor comes with a unique set of features to increase the productivity of your ML team:

1. Fully managed infrastructure – The Code Editor IDE runs on fully managed infrastructure. Amazon SageMaker takes care of keeping the instances up-to date with the latest security patches and upgrades.
2. Dial resources up and down – With Code Editor, you can seamlessly change the underlying resources (e.g., instance type, EBS volume size) on which Code Editor is running. This is beneficial for developers who want to run workloads with changing compute, memory and storage needs.
3. SageMaker provided images – Code Editor is preconfigured with the SageMaker Distribution as the default image. This container image has all the most popular ML frameworks supported by SageMaker, along with SageMaker Python SDK, boto3, and other AWS and data science specific libraries installed. This significantly reduces the time you spend setting up your environment and decreases the complexity of managing package dependencies in your ML project.
4. Amazon CodeWhisperer integration – Code Editor also comes with generative AI capabilities powered by Amazon CodeWhisperer. This native integration enables you to boost your productivity by generating code suggestions within the IDE.
5. Integration with other AWS services – You get native integration with Amazon Simple Storage Service (S3) buckets, Amazon Elastic Container Registry (ECR) repositories, Amazon RedShift, Amazon CloudWatch, and more via the AWS Toolkit for VS Code which simplifies development in cloud.

Architecture details

When launching Code Editor in SageMaker Studio, you’re creating a new application that runs as a container in an EC2 instance of the type you selected when configuring your Code Editor space. SageMaker Studio handles the provisioning of underlying resources for you in a service managed account. The following diagram depicts a simplified version of the Code Editor IDE application architecture:

For a given user profile, you can launch multiple Code Editor spaces, with a variety of ML instance types (including accelerated computing instances). Each space defines the attached EBS volume size, the instance type and the type of application to run in the space (for example, Code Editor). When users run the space, the underlying EC2 instance is provisioned and a SageMaker Studio Code Editor app is instantiated, based on the selected container image. The EBS volume is persisted across Start/Stop cycles of the IDE app. If users stop the Code Editor app (for example, to save on compute costs), the compute resources are stopped but the EBS volume is preserved and re-attached to the instance at restart.

All Code Editor applications run isolated; if you need to share data across applications, you can attach a shared Amazon Elastic File System (EFS) drive.

In order for your Code Editor IDE to use the pre-installed AWS Toolkit extension for VS Code and use integrated AWS services such as Amazon CodeWhisperer or data sources such as Amazon S3 and Amazon Redshift you have to make sure that:

• Your SageMaker Studio user profile’s execution role has appropriate permissions to use the services you want to work with.
• You have a way to communicate to those services in case you have a VPC-only mode SageMaker Studio domain. For more details on the requirements to use AWS services in a VPC-only mode Studio domain, refer to Connect SageMaker Studio Notebooks in a VPC to External Resources.

Solution overview

In the following sections, we share how you can develop an example ML project with Code Editor on Amazon SageMaker Studio. We will deploy a Mistral-7B large language model (LLM) model into an Amazon SageMaker real-time endpoint using a built-in container from HuggingFace. In this example, Code Editor can be used by an ML engineering team who needs advanced IDE features to debug their code and deploy the endpoint. You can find the sample code in this GitHub repo. We show how you can structure your code for easy collaboration between team members, how you can use the AWS Toolkit for VS Code and Amazon Code Whisperer to speed up your development, and how to deploy the Mistral-7B model on a SageMaker endpoint. Let’s walk through some of the common developer tasks in the IDE.

Interacting with AWS services directly from your IDE

Out of the box, Code Editor comes with the AWS Toolkit for Visual Studio Code to provide you with an integrated experience to other AWS services during your project. Based on your SageMaker Studio user profile AWS Identity and Access Management (IAM) permission, you can interact with data in your Amazon S3 buckets, find container images in Amazon ECR, visualize Amazon CloudWatch logs for your SageMaker endpoint, and take advantage of other features to run your end-to-end ML project from your IDE.

Structure your code repository for effortless collaboration

You can structure your project repository to maximize the productivity of your team. For example, you can setup a single repository, aiming to strike a balance between common Python project conventions and your team collaboration needs.

Your code repository can contain a .vscode folder with all the necessary files for standardizing dependencies, extensions, and configurations across the different team members. Refer to the following animation for reference.

You can share dependencies across team members through a requirements.txt file. You can also specify a config.yaml file to share the launch primitives for your SageMaker endpoint. Your Code Editor session will share the same dependencies and config as your team members, and allow you to quickly develop and debug you inference code and endpoint

Develop and debug your code in the IDE

In the following example, we show how you can develop and debug your inference.py script that will be used in your SageMaker endpoint:

Generate code and test cases with Amazon CodeWhisperer

As part of the AWS Toolkit in your Code Editor, Amazon CodeWhisperer allows you to build faster and more securely with an AI coding companion. It can provide you with real-time code suggestions, is optimized for use with AWS services, and comes with built-in security scanning. In our example we use Amazon CodeWhisperer to generate whole line and full function code to deploy and test your SageMaker endpoint

Deploying your LLM into a SageMaker endpoint

You can deploy your model to a SageMaker endpoint from your IDE and monitor its status directly from SageMaker Studio.

As you scale your ML project into a production-ready application, Code Editor and the AWS Toolkit will allow you to manage and monitor your LLM application’s resources as you build, deploy, and run it.

Conclusion

Code Editor is available in all AWS Regions where Amazon SageMaker Studio is available (except GovCloud), and you only pay for the underlying compute and storage resources within SageMaker or other AWS services, based on your usage.

To get started with Code Editor on Amazon SageMaker Studio, you can use the AWS Free Tier, with 250 hours of ml.t3.medium instance on Amazon SageMaker Studio per month for the first 2 months. For more details, refer to Amazon SageMaker Pricing.

About the Authors

Eric Peña is a Senior Technical Product Manager in the AWS Artificial Intelligence Platforms team, working on Amazon SageMaker Interactive Machine Learning. He currently focuses on IDE integrations on SageMaker Studio. He holds an MBA degree from MIT Sloan and outside of work enjoys playing basketball and football.

Vikesh Pandey is a Machine Learning Specialist Solutions Architect at AWS, helping customers from financial industries design and build solutions on generative AI and ML. Outside of work, Vikesh enjoys trying out different cuisines and playing outdoor sports.

Bruno Pistone is an AI/ML Specialist Solutions Architect for AWS based in Milan. He works with large customers helping them to deeply understand their technical needs and design AI and Machine Learning solutions that make the best use of the AWS Cloud and the Amazon Machine Learning stack. His expertise include: Machine Learning end to end, Machine Learning Industrialization, and Generative AI. He enjoys spending time with his friends and exploring new places, as well as travelling to new destinations.

Giuseppe Angelo Porcelli is a Principal Machine Learning Specialist Solutions Architect for Amazon Web Services. With several years software engineering and an ML background, he works with customers of any size to understand their business and technical needs and design AI and ML solutions that make the best use of the AWS Cloud and the Amazon Machine Learning stack. He has worked on projects in different domains, including MLOps, computer vision, and NLP, involving a broad set of AWS services. In his free time, Giuseppe enjoys playing football.

Sofian Hamiti is an AI/ML specialist Solutions Architect at AWS. He helps customers across industries accelerate their AI/ML journey by helping them build and operationalize end-to-end machine learning solutions.

Read More

As democratization of foundation models (FMs) becomes more prevalent and demand for AI-augmented services increases, software as a service (SaaS) providers are looking to use machine learning (ML) platforms that support multiple tenants—for data scientists internal to their organization and external customers. More and more companies are realizing the value of using FMs to generate highly personalized and effective content for their customers. Fine-tuning FMs on your own data can significantly boost model accuracy for your specific use case, whether it be sales email generation using page visit context, generating search answers tailored to a company’s services, or automating customer support by training on historical conversations.

Providing generative AI model hosting as a service enables any organization to easily integrate, pilot test, and deploy FMs at scale in a cost-effective manner, without needing in-house AI expertise. This allows companies to experiment with AI use cases like hyper-personalized sales and marketing content, intelligent search, and customized customer service workflows. By using hosted generative models fine-tuned on trusted customer data, businesses can deliver the next level of personalized and effective AI applications to better engage and serve their customers.

Amazon SageMaker offers different ML inference options, including real-time, asynchronous, and batch transform. This post focuses on providing prescriptive guidance on hosting FMs cost-effectively at scale. Specifically, we discuss the quick and responsive world of real-time inference, exploring different options for real-time inference for FMs.

For inference, multi-tenant AI/ML architectures need to consider the requirements for data and models, as well as the compute resources that are required to perform inference from these models. It’s important to consider how multi-tenant AI/ML models are deployed—ideally, in order to optimally utilize CPUs and GPUs, you have to be able to architect an inferencing solution that can enhance serving throughput and reduce cost by ensuring that models are distributed across the compute infrastructure in an efficient manner. In addition, customers are looking for solutions that help them deploy a best-practice inferencing architecture without needing to build everything from scratch.

SageMaker Inference is a fully managed ML hosting service. It supports building generative AI applications while meeting regulatory standards like FedRAMP. SageMaker enables cost-efficient scaling for high-throughput inference workloads. It supports diverse workloads including real-time, asynchronous, and batch inferences on hardware like AWS Inferentia, AWS Graviton, NVIDIA GPUs, and Intel CPUs. SageMaker gives you full control over optimizations, workload isolation, and containerization. It enables you to build generative AI as a service solution at scale with support for multi-model and multi-container deployments.

Challenges of hosting foundation models at scale

The following are some of the challenges in hosting FMs for inference at scale:

• Large memory footprint – FMs with tens or hundreds of billions of model parameters often exceed the memory capacity of a single accelerator chip.
• Transformers are slow – Autoregressive decoding in FMs, especially with long input and output sequences, exacerbates memory I/O operations. This culminates in unacceptable latency periods, adversely affecting real-time inference.
• Cost – FMs necessitate ML accelerators that provide both high memory and high computational power. Achieving high throughput and low latency without sacrificing either is a specialized task, requiring a deep understanding of hardware-software acceleration co-optimization.
• Longer time-to-market – Optimal performance from FMs demands rigorous tuning. This specialized tuning process, coupled with the complexities of infrastructure management, results in elongated time-to-market cycles.
• Workload isolation – Hosting FMs at scale introduces challenges in minimizing the blast-radius and handling noisy neighbors. The ability to scale each FM in response to model-specific traffic patterns requires heavy lifting.
• Scaling to hundreds of FMs – Operating hundreds of FMs simultaneously introduces substantial operational overhead. Effective endpoint management, appropriate slicing and accelerator allocation, and model-specific scaling are tasks that compound in complexity as more models are deployed.

Fitness functions

Deciding on the right hosting option is important because it impacts the end-users rendered by your applications. For this purpose, we’re borrowing the concept of fitness functions, which was coined by Neal Ford and his colleagues from AWS Partner Thought Works in their work Building Evolutionary Architectures. Fitness functions provide a prescriptive assessment of various hosting options based on your objectives. Fitness functions help you obtain the necessary data to allow for the planned evolution of your architecture. They set measurable values to assess how close your solution is to achieving your set goals. Fitness functions can and should be adapted as the architecture evolves to guide a desired change process. This provides architects with a tool to guide their teams while maintaining team autonomy.

We propose considering the following fitness functions when it comes to selecting the right FM inference option at scale and cost-effectively:

• Foundation model size – FMs are based on transformers. Transformers are slow and memory-hungry on generating long text sequences due to the sheer size of the models. Large language models (LLMs) are a type of FM that, when used to generate text sequences, need immense amounts of computing power and have difficulty accessing the available high bandwidth memory (HBM) and compute capacity. This is because a large portion of the available memory bandwidth is consumed by loading the model’s parameters and by the auto-regressive decoding process. As a result, even with massive amounts of compute power, FMs are limited by memory I/O and computation limits. Therefore, model size determines a lot of decisions, such as whether the model will fit on a single accelerator or require multiple ML accelerators using model sharding on the instance to run the inference at a higher throughput. Models with more than 3 billion parameters will generally start requiring multiple ML accelerators because the model might not fit into a single accelerator device.
• Performance and FM inference latency – Many ML models and applications are latency critical, in which the inference latency must be within the bounds specified by a service-level objective. FM inference latency depends on a multitude of factors, including:
  • FM model size – Model size, including quantization at runtime.
  • Hardware – Compute (TFLOPS), HBM size and bandwidth, network bandwidth, intra-instance interconnect speed, and storage bandwidth.
  • Software environment – Model server, model parallel library, model optimization engine, collective communication performance, model network architecture, quantization, and ML framework.
  • Prompt – Input and output length and hyperparameters.
  • Scaling latency – Time to scale in response to traffic.
  • Cold start latency – Features like pre-warming the model load can reduce the cold start latency in loading the FM.
• Workload isolation – This refers to workload isolation requirements from a regulatory and compliance perspective, including protecting confidentiality and integrity of AI models and algorithms, confidentiality of data during AI inference, and protecting AI intellectual property (IP) from unauthorized access or from a risk management perspective. For example, you can reduce the impact of a security event by purposefully reducing the blast-radius or by preventing noisy neighbors.
• Cost-efficiency – To deploy and maintain an FM model and ML application on a scalable framework is a critical business process, and the costs may vary greatly depending on choices made about model hosting infrastructure, hosting option, ML frameworks, ML model characteristics, optimizations, scaling policy, and more. The workloads must utilize the hardware infrastructure optimally to ensure that the cost remains in check. This fitness function specifically refers to the infrastructure cost, which is part of the overall total cost of ownership (TCO). The infrastructure costs are the combined costs for storage, network, and compute. It’s also critical to understand other components of TCO, including operational costs and security and compliance costs. Operational costs are the combined costs of operating, monitoring, and maintaining the ML infrastructure. The operational costs are calculated as the number of engineers required based on each scenario and the annual salary of engineers, aggregated over a specific period. They automatically scale to zero per model when there’s no traffic to save costs.
• Scalability – This includes:
  • Operational overhead in managing hundreds of FMs for inference in a multi-tenant platform.
  • The ability to pack multiple FMs in a single endpoint and scale per model.
  • Enabling instance-level and model container-level scaling based on workload patterns.
  • Support for scaling to hundreds of FMs per endpoint.
  • Support for the initial placement of the models in the fleet and handling insufficient accelerators.

Representing the dimensions in fitness functions

We use a spider chart, also sometimes called a radar chart, to represent the dimensions in the fitness functions. A spider chart is often used when you want to display data across several unique dimensions. These dimensions are usually quantitative, and typically range from zero to a maximum value. Each dimension’s range is normalized to one another, so that when we draw our spider chart, the length of a line from zero to a dimension’s maximum value will be the same for every dimension.

The following chart illustrates the decision-making process involved when choosing your architecture on SageMaker. Each radius on the spider chart is one of the fitness functions that you will prioritize when you build your inference solution.

Ideally, you’d like a shape that is equilateral across all sides (a pentagon). That shows that you are able to optimize across all fitness functions. But the reality is that it will be challenging to achieve that shape—as you prioritize one fitness function, it will affect the lines for the other radius. This means there will always be trade-offs depending on what is most important for your generative AI application, and you’ll have a graph that will be skewed towards a specific radius. This is the criteria that you may be willing to de-prioritize in favor of the others depending on how you view each function. In our chart, each fitness function’s metric weight is defined as such—the lower the value, the less optimal it is for that fitness function (with the exception of model size, in which case the higher the value, the larger the size of the model).

For example, let’s take a use case where you would like to use a large summarization model (such as Anthropic Claude) to create work summaries of service cases and customer engagements based on case data and customer history. We have the following spider chart.

Because this may involve sensitive customer data, you’re choosing to isolate this workload from other models and host it on a single-model endpoint, which can make it challenging to scale because you have to spin up and manage separate endpoints for each FM. The generative AI application you’re using the model with is being used by service agents in real time, so latency and throughput are a priority, hence the need to use larger instance types, such as a P4De. In this situation, the cost may have to be higher because the priority is isolation, latency, and throughput.

Another use case would be a service organization building a Q&A chatbot application that is customized for a large number of customers. The following spider chart reflects their priorities.

Each chatbot experience may need to be tailored to each specific customer. The models being used may be relatively smaller (FLAN-T5-XXL, Llama 7B, and k-NN), and each chatbot operates at a designated set of hours for different time zones each day. The solution may also have Retrieval Augmented Generation (RAG) incorporated with a database containing all the knowledge base items to be used with inference in real time. There isn’t any customer-specific data being exchanged through this chatbot. Cold start latencies are tolerable because the chatbots operate on a defined schedule. For this use case, you can choose a multi-model endpoint architecture, and may be able minimize cost by using smaller instance types (like a G5) and potentially reduce operational overhead by hosting multiple models on each endpoint at scale. With the exception of workload isolation, fitness functions in this use case may have more of an even priority, and trade-offs are minimized to an extent.

One final example would be an image generation application using a model like Stable Diffusion 2.0, which is a 3.5-billion-parameter model. Our spider chart is as follows.

This is a subscription-based application serving thousands of FMs and customers. The response time needs to be quick because each customer expects a fast turnaround of image outputs. Throughput is critical as well because there will be hundreds of thousands of requests at any given second, so the instance type will have to be a larger instance type, like a P4D that has enough GPU and memory. For this you can consider building a multi-container endpoint hosting multiple copies of the model to denoise image generation from one request set to another. For this use case, in order to prioritize latency and throughput and accommodate user demand, cost of compute and workload isolation will be the trade-offs.

Applying fitness functions to selecting the FM hosting option

In this section, we show you how to apply the preceding fitness functions in selecting the right FM hosting option on SageMaker FMs at scale.

SageMaker single-model endpoints

SageMaker single-model endpoints allow you to host one FM on a container hosted on dedicated instances for low latency and high throughput. These endpoints are fully managed and support auto scaling. You can configure the single-model endpoint as a provisioned endpoint where you pass in endpoint infrastructure configuration such as the instance type and count, where SageMaker automatically launches compute resources and scales them in and out depending on the auto scaling policy. You can scale to hosting hundreds of models using multiple single-model endpoints and employ a cell-based architecture for increased resiliency and reduced blast-radius.

When evaluating fitness functions for a provisioned single-model endpoint, consider the following:

• Foundation model size – This is suitable if you have models that can’t fit into single ML accelerator’s memory and therefore need multiple accelerators in an instance.
• Performance and FM inference latency – This is relevant for latency-critical generative AI applications.
• Workload isolation – Your application may need Amazon Elastic Compute Cloud (Amazon EC2) instance-level isolation due to security compliance reasons. Each FM will get a separate inference endpoint and won’t share the EC2 instance with another other model. For example, you can isolate a HIPAA-related model inference workload (such as a PHI detection model) in a separate endpoint with a dedicated security group configuration with network isolation. You can isolate your GPU-based model inference workload from others based on Nitro-based EC2 instances like p4dn in order to isolate them from less trusted workloads. The Nitro System-based EC2 instances provide a unique approach to virtualization and isolation, enabling you to secure and isolate sensitive data processing from AWS operators and software at all times. It provides the most important dimension of confidential computing as an intrinsic, on-by-default set of protections from the system software and cloud operators. This option also supports deploying AWS Marketplace models provided by third-party model providers on SageMaker.

SageMaker multi-model endpoints

SageMaker multi-model endpoints (MMEs) allow you to co-host multiple models on a GPU core, share GPU instances behind an endpoint across multiple models, and dynamically load and unload models based on the incoming traffic. With this, you can significantly save cost and achieve the best price-performance.

MMEs are the best choice if you need to host smaller models that can all fit into a single ML accelerator on an instance. This strategy should be considered if you have a large number (up to thousands) of similar sized (fewer than 1 billion parameters) models that you can serve through a shared container within an instance and don’t need to access all the models at the same time. You can load the model that needs to be used and then unload it for a different model.

MMEs are also designed for co-hosting models that use the same ML framework because they use the shared container to load multiple models. Therefore, if you have a mix of ML frameworks in your model fleet (such as PyTorch and TensorFlow), a SageMaker endpoint with InferenceComponents is a better choice. We discuss InferenceComponents more later in this post.

Finally, MMEs are suitable for applications that can tolerate an occasional cold start latency penalty because infrequently used models can be off-loaded in favor of frequently invoked models. If you have a long tail of infrequently accessed models, a multi-model endpoint can efficiently serve this traffic and enable significant cost savings.

Consider the following when assessing when to use MMEs:

• Foundation model size – You may have models that fit into single ML accelerator’s HBM on an instance and therefore don’t need multiple accelerators.
• Performance and FM inference latency – You may have generative AI applications that can tolerate cold start latency when the model is requested and is not in the memory.
• Workload isolation – Consider having all the models share the same container.
• Scalability – Consider the following:
  • You can pack multiple models in a single endpoint and scale per model and ML instance.
  • You can enable instance-level auto scaling based on workload patterns.
  • MMEs support scaling to thousands of models per endpoint. You don’t need to maintain per-model auto scaling and deployment configuration.
  • You can use hot deployment whenever the model is requested by the inference request.
  • You can load the models dynamically as per the inference request and unload in response to memory pressure.
  • You can time share the underlying the resources with the models.
• Cost-efficiency – Consider time sharing the resource across the models by dynamic loading and unloading of the models, resulting in cost savings.

SageMaker inference endpoint with InferenceComponents

The new SageMaker inference endpoint with InferenceComponents provides a scalable approach to hosting multiple FMs in a single endpoint and scaling per model. It provides you with fine-grained control to allocate resources (accelerators, memory, CPU) and set auto scaling policies on a per-model basis to get assured throughput and predictable performance, and you can manage the utilization of compute across multiple models individually. If you have a lot of models of varying sizes and traffic patterns that you need to host, and the model sizes don’t allow them to fit in a single accelerator’s memory, this is the best option. It also allows you to scale to zero to save costs, but your application latency requirements need to be flexible enough to account for a cold start time for models. This option allows you the most flexibility in utilizing your compute as long as container-level isolation per customer or FM is sufficient. For more details on the new SageMaker endpoint with InferenceComponents, refer to the detailed post Reduce model deployment costs by 50% on average using the latest features of Amazon SageMaker.

Consider the following when determining when you should use an endpoint with InferenceComponents:

• Foundation model size – This is suitable for models that can’t fit into single ML accelerator’s memory and therefore need multiple accelerators in an instance.
• Performance and FM inference latency – This is suitable for latency-critical generative AI applications.
• Workload isolation – You may have applications where container-level isolation is sufficient.
• Scalability – Consider the following:
  • You can pack multiple FMs in a single endpoint and scale per model.
  • You can enable instance-level and model container-level scaling based on workload patterns.
  • This method supports scaling to hundreds of FMs per endpoint. You don’t need to configure the auto scaling policy for each model or container.
  • It supports the initial placement of the models in the fleet and handling insufficient accelerators.
• Cost-efficiency – You can scale to zero per model when there is no traffic to save costs.

Packing multiple FMs on same endpoint: Model grouping

Determining what inference architecture strategy you employ on SageMaker depends on your application priorities and requirements. Some SaaS providers are selling into regulated environments that impose strict isolation requirements—they need to have an option that enables them to offer to some or all of their FMs the option of being deployed in a dedicated model. But in order to optimize costs and gain economies of scale, SaaS providers need to also have multi-tenant environments where they host multiple FMs across a shared set of SageMaker resources. Most organizations will probably have a hybrid hosting environment where they have both single-model endpoints and multi-model or multi-container endpoints as part of their SageMaker architecture.

A critical exercise you will need to perform when architecting this distributed inference environment is to group your models for each type of architecture, you’ll need to set up in your SageMaker endpoints. The first decision you’ll have to make is around workload isolation requirements—you will need to isolate the FMs that need to be in their own dedicated endpoints, whether it’s for security reasons, reducing the blast-radius and noisy neighbor risk, or meeting strict SLAs for latency.

Secondly, you’ll need to determine whether the FMs fit into a single ML accelerator or require multiple accelerators, what the model sizes are, and what their traffic patterns are. Similar sized models that collectively serve to support a central function could logically be grouped together by co-hosting multiple models on an endpoint, because these would be part of a single business application that is managed by a central team. For co-hosting multiple models on the same endpoint, a grouping exercise needs to be performed to determine which models can sit in a single instance, a single container, or multiple containers.

Grouping the models for MMEs

MMEs are best suited for smaller models (fewer than 1 billion parameters that can fit into single accelerator) and are of similar in size and invocation latencies. Some variation in model size is acceptable; for example, Zendesk’s models range from 10–50 MB, which works fine, but variations in size that are a factor of 10, 50, or 100 times greater aren’t suitable. Larger models may cause a higher number of loads and unloads of smaller models to accommodate sufficient memory space, which can result in added latency on the endpoint. Differences in performance characteristics of larger models could also consume resources like CPU unevenly, which could impact other models on the instance.

The models that are grouped together on the MME need to have staggered traffic patterns to allow you to share compute across the models for inference. Your access patterns and inference latency also need to allow for some cold start time as you switch between models.

The following are some of the recommended criteria for grouping the models for MMEs:

• Smaller models – Use models with fewer than 1 billion parameters
• Model size – Group similar sized models and co-host into the same endpoint
• Invocation latency – Group models with similar invocation latency requirements that can tolerate cold starts
• Hardware – Group the models using the same underlying EC2 instance type

Grouping the models for an endpoint with InferenceComponents

A SageMaker endpoint with InferenceComponents is best suited for hosting larger FMs (over 1 billion parameters) at scale that require multiple ML accelerators or devices in an EC2 instance. This option is suited for latency-sensitive workloads and applications where container-level isolation is sufficient. The following are some of the recommended criteria for grouping the models for an endpoint with multiple InferenceComponents:

• Hardware – Group the models using the same underlying EC2 instance type
• Model size – Grouping the model based on model size is recommended but not mandatory

Summary

In this post, we looked at three real-time ML inference options (single endpoints, multi-model endpoints, and endpoints with InferenceComponents) in SageMaker to efficiently host FMs at scale cost-effectively. You can use the five fitness functions to help you choose the right SageMaker hosting option for FMs at scale. Group the FMs and co-host them on SageMaker inference endpoints using the recommended grouping criteria. In addition to the fitness functions we discussed, you can use the following table to decide which shared SageMaker hosting option is best for your use case. You can find code samples for each of the FM hosting options on SageMaker in the following GitHub repos: single SageMaker endpoint, multi-model endpoint, and InferenceComponents endpoint.

About the authors

Mehran Najafi, PhD, is a Senior Solutions Architect for AWS focused on AI/ML and SaaS solutions at Scale.

Dhawal Patel is a Principal Machine Learning Architect at AWS. He has worked with organizations ranging from large enterprises to mid-sized startups on problems related to distributed computing, and Artificial Intelligence. He focuses on Deep learning including NLP and Computer Vision domains. He helps customers achieve high performance model inference on SageMaker.

Rielah DeJesus is a Principal Solutions Architect at AWS who has successfully helped various enterprise customers in the DC, Maryland, and Virginia area move to the cloud. A customer advocate and technical advisor, she helps organizations like Heroku/Salesforce achieve success on the AWS platform. She is a staunch supporter of Women in IT and very passionate about finding ways to creatively use technology and data to solve everyday challenges.

Read More

As organizations deploy models to production, they are constantly looking for ways to optimize the performance of their foundation models (FMs) running on the latest accelerators, such as AWS Inferentia and GPUs, so they can reduce their costs and decrease response latency to provide the best experience to end-users. However, some FMs don’t fully utilize the accelerators available with the instances they’re deployed on, leading to an inefficient use of hardware resources. Some organizations deploy multiple FMs to the same instance to better utilize all of the available accelerators, but this requires complex infrastructure orchestration that is time consuming and difficult to manage. When multiple FMs share the same instance, each FM has its own scaling needs and usage patterns, making it challenging to predict when you need to add or remove instances. For example, one model may be used to power a user application where usage can spike during certain hours, whereas another model may have a more consistent usage pattern. In addition to optimizing costs, customers want to provide the best end-user experience by reducing latency. To do this, they often deploy multiple copies of a FM to field requests from users in parallel. Because FM outputs could range from a single sentence to multiple paragraphs, the time it takes to complete the inference request varies significantly, leading to unpredictable spikes in latency if the requests are routed randomly between instances. Amazon SageMaker now supports new inference capabilities that help you reduce deployment costs and latency.

You can now create inference component-based endpoints and deploy machine learning (ML) models to a SageMaker endpoint. An inference component (IC) abstracts your ML model and enables you to assign CPUs, GPU, or AWS Neuron accelerators, and scaling policies per model. Inference components offer the following benefits:

• SageMaker will optimally place and pack models onto ML instances to maximize utilization, leading to cost savings.
• SageMaker will scale each model up and down based on your configuration to meet your ML application requirements.
• SageMaker will scale to add and remove instances dynamically to ensure capacity is available while keeping idle compute to a minimum.
• You can scale down to zero copies of a model to free up resources for other models. You can also specify to keep important models always loaded and ready to serve traffic.

With these capabilities, you can reduce model deployment costs by 50% on average. The cost savings will vary depending on your workload and traffic patterns. Let’s take a simple example to illustrate how packing multiple models on a single endpoint can maximize utilization and save costs. Let’s say you have a chat application that helps tourists understand local customs and best practices built using two variants of Llama 2: one fine-tuned for European visitors and the other fine-tuned for American visitors. We expect traffic for the European model between 00:01–11:59 UTC and the American model between 12:00–23:59 UTC. Instead of deploying these models on their own dedicated instances where they will sit idle half the time, you can now deploy them on a single endpoint to save costs. You can scale down the American model to zero when it isn’t needed to free up capacity for the European model and vice versa. This allows you to utilize your hardware efficiently and avoid waste. This is a simple example using two models, but you can easily extend this idea to pack hundreds of models onto a single endpoint that automatically scales up and down with your workload.

In this post, we show you the new capabilities of IC-based SageMaker endpoints. We also walk you through deploying multiple models using inference components and APIs. Lastly, we detail some of the new observability capabilities and how to set up auto scaling policies for your models and manage instance scaling for your endpoints. You can also deploy models through our new simplified, interactive user experience. We also support advanced routing capabilities to optimize the latency and performance of your inference workloads.

Building blocks

Let’s take a deeper look and understand how these new capabilities work. The following is some new terminology for SageMaker hosting:

• Inference component – A SageMaker hosting object that you can use to deploy a model to an endpoint. You can create an inference component by supplying the following:
  • The SageMaker model or specification of a SageMaker-compatible image and model artifacts.
  • Compute resource requirements, which specify the needs of each copy of your model, including CPU cores, host memory, and number of accelerators.
• Model copy – A runtime copy of an inference component that is capable of serving requests.
• Managed instance auto scaling – A SageMaker hosting capability to scale up or down the number of compute instances used for an endpoint. Instance scaling reacts to the scaling of inference components.

To create a new inference component, you can specify a container image and a model artifact, or you can use SageMaker models that you may have already created. You also need to specify the compute resource requirements such as the number of host CPU cores, host memory, or the number of accelerators your model needs to run.

When you deploy an inference component, you can specify MinCopies to ensure that the model is already loaded in the quantity that you require, ready to serve requests.

You also have the option to set your policies so that inference component copies scale to zero. For example, if you have no load running against an IC, the model copy will be unloaded. This can free up resources that can be replaced by active workloads to optimize the utilization and efficiency of your endpoint.

As inference requests increase or decrease, the number of copies of your ICs can also scale up or down based on your auto scaling policies. SageMaker will handle the placement to optimize the packing of your models for availability and cost.

In addition, if you enable managed instance auto scaling, SageMaker will scale compute instances according to the number of inference components that need to be loaded at a given time to serve traffic. SageMaker will scale up the instances and pack your instances and inference components to optimize for cost while preserving model performance. Although we recommend the use of managed instance scaling, you also have the option to manage the scaling yourself, should you choose to, through application auto scaling.

SageMaker will rebalance inference components and scale down the instances if they are no longer needed by inference components and save your costs.

Walkthrough of APIs

SageMaker has introduced a new entity called the InferenceComponent. This decouples the details of hosting the ML model from the endpoint itself. The InferenceComponent allows you to specify key properties for hosting the model like the SageMaker model you want to use or the container details and model artifacts. You also specify number of copies of the components itself to deploy, and number of accelerators (GPUs, Inf, or Trn accelerators) or CPU (vCPUs) required. This provides more flexibility for you to use a single endpoint for any number of models you plan to deploy to it in the future.

Let’s look at the Boto3 API calls to create an endpoint with an inference component. Note that there are some parameters that we address later in this post.

The following is example code for CreateEndpointConfig:

sagemaker_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ExecutionRoleArn=role,
    ProductionVariants=[{
        "VariantName": variant_name,
        "InstanceType": instance_type,
        "InitialInstanceCount": initial_instance_count,
        "ModelDataDownloadTimeoutInSeconds": model_data_download_timeout_in_seconds,
        "ContainerStartupHealthCheckTimeoutInSeconds": container_startup_health_check_timeout_in_seconds,
        {"ManagedInstanceScaling": {
            "Status": "ENABLED",
            "MinInstanceCount": initial_instance_count,
            "MaxInstanceCount": max_instance_count,
            }
        },
    }],
)

The following is example code for CreateEndpoint:

sagemaker_client.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=endpoint_config_name,
)

The following is example code for CreateInferenceComponent:

sm_client.create_inference_component(
    InferenceComponentName=inference_component_name,
    EndpointName=endpoint_name,
    VariantName=variant_name,
    Specification={
        "Container": {
            "Image": inference_image_uri,
            "ArtifactUrl": s3_code_artifact,
        },
        "StartupParameters": {
            "ModelDataDownloadTimeoutInSeconds": 300,
            "ContainerStartupHealthCheckTimeoutInSeconds": 600,
        },
        "ComputeResourceRequirements": {"NumberOfAcceleratorDevicesRequired": 1, "MinMemoryRequiredInMb": 1024}
    },
    RuntimeConfig={"CopyCount": 1},
)

This decoupling of InferenceComponent to an endpoint provides flexibility. You can host multiple models on the same infrastructure, adding or removing them as your requirements change. Each model can be updated independently as needed. Additionally, you can scale models according to your business needs. InferenceComponent also allows you to control capacity per model. In other words, you can determine how many copies of each model to host. This predictable scaling helps you meet the specific latency requirements for each model. Overall, InferenceComponent gives you much more control over your hosted models.

In the following table, we show a side-by-side comparison of the high-level approach to creating and invoking an endpoint without InferenceComponent and with InferenceComponent. Note that CreateModel() is now optional for IC-based endpoints.

The introduction of InferenceComponent allows you to scale at a model level. See Delve into instance and IC auto scaling for more details on how InferenceComponent works with auto scaling.

When invoking the SageMaker endpoint, you can now specify the new parameter InferenceComponentName to hit the desired InferenceComponentName. SageMaker will handle routing the request to the instance hosting the requested InferenceComponentName. See the following code:

smr_client = boto3.client("sagemaker-runtime") 
response_model = smr_client.invoke_endpoint( 
    InferenceComponentName=inference_component_name,
    EndpointName=endpoint_name, 
    Body=payload, 
    ContentType="application/json", )

By default, SageMaker uses random routing of the requests to the instances backing your endpoint. If you want to enable least outstanding requests routing, you can set the routing strategy in the endpoint config’s RoutingConfig:

sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ExecutionRoleArn=role,
    ProductionVariants=[{
        "VariantName": variant_name,
        "InstanceType": instance_type,
        "InitialInstanceCount": initial_instance_count,
        ...
        'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'
            }
    }],
)

Least outstanding requests routing routes to the specific instances that have more capacity to process requests. This will provide more uniform load-balancing and resource utilization.

In addition to CreateInferenceComponent, the following APIs are now available:

• DescribeInferenceComponent
• DeleteInferenceComponent
• UpdateInferenceComponent
• ListInferenceComponents

InferenceComponent logs and metrics

InferenceComponent logs are located in /aws/sagemaker/InferenceComponents/<InferenceComponentName>. All logs sent to stderr and stdout in the container are sent to these logs in Amazon CloudWatch.

With the introduction of IC-based endpoints, you now have the ability to view additional instance metrics, inference component metrics, and invocation metrics.

For SageMaker instances, you can now track the GPUReservation and CPUReservation metrics to see the resources reserved for an endpoint based on the inference components that you have deployed. These metrics can help you size your endpoint and auto scaling policies. You can also view the aggregate metrics associated with all models deployed to an endpoint.

SageMaker also exposes metrics at an inference component level, which can show a more granular view of the utilization of resources for the inference components that you have deployed. This allows you to get a view of how much aggregate resource utilization such as GPUUtilizationNormalized and GPUMemoryUtilizationNormalized for each inference component you have deployed that may have zero or many copies.

Lastly, SageMaker provides invocation metrics, which now tracks invocations for inference components aggregately (Invocations) or per copy instantiated (InvocationsPerCopy)

For a comprehensive list of metrics, refer to SageMaker Endpoint Invocation Metrics.

Model-level auto scaling

To implement the auto scaling behavior we described, when creating the SageMaker endpoint configuration and inference component, you define the initial instance count and initial model copy count, respectively. After you create the endpoint and corresponding ICs, to apply auto scaling at the IC level, you need to first register the scaling target and then associate the scaling policy to the IC.

When implementing the scaling policy, we use SageMakerInferenceComponentInvocationsPerCopy, which is a new metric introduced by SageMaker. It captures the average number of invocations per model copy per minute.

aas_client.put_scaling_policy(
    PolicyName=endpoint_name,
    PolicyType='TargetTrackingScaling',
    ServiceNamespace=service_namespace,
    ResourceId=resource_id,
    ScalableDimension=scalable_dimension,
    TargetTrackingScalingPolicyConfiguration={
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerInferenceComponentInvocationsPerCopy",
        },
        "TargetValue": autoscaling_target_value,
        "ScaleInCooldown": 300,  # default
        "ScaleOutCooldown": 300,  # default
    },
)

After you set the scaling policy, SageMaker creates two CloudWatch alarms for each autoscaling target: one to trigger scale-out if in alarm for 3 minutes (three 1-minute data points) and one to trigger scale-in if in alarm for 15 minutes (15 1-minute data points), as shown in the following screenshot. The time to trigger the scaling action is usually 1–2 minutes longer than those minutes because it takes time for the endpoint to publish metrics to CloudWatch, and it also takes time for AutoScaling to react. The cool-down period is the amount of time, in seconds, after a scale-in or scale-out activity completes before another scale-out activity can start. If the scale-out cool-down is shorter than that the endpoint update time, then it takes no effect, because it’s not possible to update a SageMaker endpoint when it is in Updating status.

Note that, when setting up IC-level auto scaling, you need to make sure the MaxInstanceCount parameter is equal to or smaller than the maximum number of ICs this endpoint can handle. For example, if your endpoint is only configured to have one instance in the endpoint configuration and this instance can only host a maximum of four copies of the model, then the MaxInstanceCount should be equal to or smaller than 4. However, you can also use the managed auto scaling capability provided by SageMaker to automatically scale the instance count based on the required model copy number to fulfil the need of more compute resources. The following code snippet demonstrates how to set up managed instance scaling during the creation of the endpoint configuration. This way, when the IC-level auto scaling requires more instance count to host the model copies, SageMaker will automatically scale out the instance number to allow the IC-level scaling to be successful.

sagemaker_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ExecutionRoleArn=role,
    ProductionVariants=[{
        "VariantName": variant_name,
        "InstanceType": instance_type,
        "InitialInstanceCount": initial_instance_count,
        "ModelDataDownloadTimeoutInSeconds": model_data_download_timeout_in_seconds,
        "ContainerStartupHealthCheckTimeoutInSeconds": container_startup_health_check_timeout_in_seconds,
        {"ManagedInstanceScaling": {
            "Status": "ENABLED",
            "MinInstanceCount": initial_instance_count,
            "MaxInstanceCount": max_instance_count,
            }
        },
    }],
)

You can apply multiple auto scaling policies against the same endpoint, which means you will be able to apply the traditional auto scaling policy to the endpoints created with ICs and scale up and down based on the other endpoint metrics. For more information, refer to Optimize your machine learning deployments with auto scaling on Amazon SageMaker. However, although this is possible, we still recommend using managed instance scaling over managing the scaling yourself.

Conclusion

In this post, we introduced a new feature in SageMaker inference that will help you maximize the utilization of compute instances, scale to hundreds of models, and optimize costs, while providing predictable performance. Furthermore, we provided a walkthrough of the APIs and showed you how to configure and deploy inference components for your workloads.

We also support advanced routing capabilities to optimize the latency and performance of your inference workloads. SageMaker can help you optimize your inference workloads for cost and performance and give you model-level granularity for management. We have created a set of notebooks that will show you how to deploy three different models, using different containers and applying auto scaling policies in GitHub. We encourage you to start with notebook 1 and get hands on with the new SageMaker hosting capabilities today!

About the authors

James Park is a Solutions Architect at Amazon Web Services. He works with Amazon.com to design, build, and deploy technology solutions on AWS, and has a particular interest in AI and machine learning. In h is spare time he enjoys seeking out new cultures, new experiences,  and staying up to date with the latest technology trends. You can find him on LinkedIn.

Melanie Li, PhD, is a Senior AI/ML Specialist TAM at AWS based in Sydney, Australia. She helps enterprise customers build solutions using state-of-the-art AI/ML tools on AWS and provides guidance on architecting and implementing ML solutions with best practices. In her spare time, she loves to explore nature and spend time with family and friends.

Marc Karp is an ML Architect with the Amazon SageMaker Service team. He focuses on helping customers design, deploy, and manage ML workloads at scale. In his spare time, he enjoys traveling and exploring new places.

Alan Tan is a Senior Product Manager with SageMaker, leading efforts on large model inference. He’s passionate about applying machine learning to the area of analytics. Outside of work, he enjoys the outdoors.

Raghu Ramesha is a Senior ML Solutions Architect with the Amazon SageMaker Service team. He focuses on helping customers build, deploy, and migrate ML production workloads to SageMaker at scale. He specializes in machine learning, AI, and computer vision domains, and holds a master’s degree in Computer Science from UT Dallas. In his free time, he enjoys traveling and photography.

Rupinder Grewal is a Sr Ai/ML Specialist Solutions Architect with AWS. He currently focuses on serving of models and MLOps on SageMaker. Prior to this role he has worked as Machine Learning Engineer building and hosting models. Outside of work he enjoys playing tennis and biking on mountain trails.

Dhawal Patel is a Principal Machine Learning Architect at AWS. He has worked with organizations ranging from large enterprises to mid-sized startups on problems related to distributed computing, and Artificial Intelligence. He focuses on Deep learning including NLP and Computer Vision domains. He helps customers achieve high performance model inference on SageMaker.

Saurabh Trikande is a Senior Product Manager for Amazon SageMaker Inference. He is passionate about working with customers and is motivated by the goal of democratizing machine learning. He focuses on core challenges related to deploying complex ML applications, multi-tenant ML models, cost optimizations, and making deployment of deep learning models more accessible. In his spare time, Saurabh enjoys hiking, learning about innovative technologies, following TechCrunch and spending time with his family.

Lakshmi Ramakrishnan is a Principal Engineer at Amazon SageMaker Machine Learning (ML) platform team in AWS, providing technical leadership for the product. He has worked in several engineering roles in Amazon for over 9 years. He has a Bachelor of Engineering degree in Information Technology from National Institute of Technology, Karnataka, India and a Master of Science degree in Computer Science from the University of Minnesota Twin Cities.

David Nigenda is a Senior Software Development Engineer on the Amazon SageMaker team, currently working on improving production machine learning workflows, as well as launching new inference features. In his spare time, he tries to keep up with his kids.

Read More

Amazon SageMaker makes it straightforward to deploy machine learning (ML) models for real-time inference and offers a broad selection of ML instances spanning CPUs and accelerators such as AWS Inferentia. As a fully managed service, you can scale your model deployments, minimize inference costs, and manage your models more effectively in production with reduced operational burden. A SageMaker real-time inference endpoint consists of an HTTPs endpoint and ML instances that are deployed across multiple Availability Zones for high availability. SageMaker application auto scaling can dynamically adjust the number of ML instances provisioned for a model in response to changes in workload. The endpoint uniformly distributes incoming requests to ML instances using a round-robin algorithm.

When ML models deployed on instances receive API calls from a large number of clients, a random distribution of requests can work very well when there is not a lot of variability in your requests and responses. But in systems with generative AI workloads, requests and responses can be extremely variable. In these cases, it’s often desirable to load balance by considering the capacity and utilization of the instance rather than random load balancing.

In this post, we discuss the SageMaker least outstanding requests (LOR) routing strategy and how it can minimize latency for certain types of real-time inference workloads by taking into consideration the capacity and utilization of ML instances. We talk about its benefits over the default routing mechanism and how you can enable LOR for your model deployments. Finally, we present a comparative analysis of latency improvements with LOR over the default routing strategy of random routing.

SageMaker LOR strategy

By default, SageMaker endpoints have a random routing strategy. SageMaker now supports a LOR strategy, which allows SageMaker to optimally route requests to the instance that is best suited to serve that request. SageMaker makes this possible by monitoring the load of the instances behind your endpoint, and the models or inference components that are deployed on each instance.

The following interactive diagram shows the default routing policy where requests coming to the model endpoints are forwarded in a random manner to the ML instances.

The following interactive diagram shows the routing strategy where SageMaker will route the request to the instance that has the least number of outstanding requests.

In general, LOR routing works well for foundational models or generative AI models when your model responds in hundreds of milliseconds to minutes. If your model response has lower latency (up to hundreds of milliseconds), you may benefit more from random routing. Regardless, we recommend that you test and identify the best routing algorithm for your workloads.

How to set SageMaker routing strategies

SageMaker now allows you to set the RoutingStrategy parameter while creating the EndpointConfiguration for endpoints. The different RoutingStrategy values that are supported by SageMaker are:

• LEAST_OUTSTANDING_REQUESTS
• RANDOM

The following is an example deployment of a model on an inference endpoint that has LOR enabled:

1. Create the endpoint configuration by setting RoutingStrategy as LEAST_OUTSTANDING_REQUESTS:

   endpoint_config_response = sm_client.create_endpoint_config(
       EndpointConfigName=endpoint_config_name,
       ProductionVariants=[
           {
               "VariantName": "variant1",
               "ModelName": model_name,
               "InstanceType": "instance_type",
               "InitialInstanceCount": initial_instance_count,
   	…..
               "RoutingConfig": {
                   'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'}
           },
       ],
   )

2. Create the endpoint using the endpoint configuration (no change):

   create_endpoint_response = sm_client.create_endpoint(
       EndpointName="endpoint_name", 
       EndpointConfigName="endpoint_config_name"
   )

Performance results

We ran performance benchmarking to measure the end-to-end inference latency and throughput of the codegen2-7B model hosted on ml.g5.24xl instances with default routing and smart routing endpoints. The CodeGen2 model belongs to the family of autoregressive language models and generates executable code when given English prompts.

In our analysis, we increased the number of ml.g5.24xl instances behind each endpoint for each test run as the number of concurrent users were increased, as shown in the following table.

We measured the end-to-end P99 latency for both endpoints and observed an 4–33% improvement in latency when the number of instances were increased from 5 to 20, as shown in the following graph.

Similarly, we observed an 15–16% improvement in the throughput per minute per instance when the number of instances were increased from 5 to 20.

This illustrates that smart routing is able to improve the traffic distribution among the endpoints, leading to improvements in end-to-end latency and overall throughput.

Conclusion

In this post, we explained the SageMaker routing strategies and the new option to enable LOR routing. We explained how to enable LOR and how it can benefit your model deployments. Our performance tests showed latency and throughput improvements during real-time inferencing. To learn more about SageMaker routing features, refer to documentation. We encourage you to evaluate your inference workloads and determine if you are optimally configured with the routing strategy.

About the Authors

James Park is a Solutions Architect at Amazon Web Services. He works with Amazon.com to design, build, and deploy technology solutions on AWS, and has a particular interest in AI and machine learning. In h is spare time he enjoys seeking out new cultures, new experiences,  and staying up to date with the latest technology trends. You can find him on LinkedIn.

Venugopal Pai is a Solutions Architect at AWS. He lives in Bengaluru, India, and helps digital-native customers scale and optimize their applications on AWS.

David Nigenda is a Senior Software Development Engineer on the Amazon SageMaker team, currently working on improving production machine learning workflows, as well as launching new inference features. In his spare time, he tries to keep up with his kids.

Deepti Ragha is a Software Development Engineer in the Amazon SageMaker team. Her current work focuses on building features to host machine learning models efficiently. In her spare time, she enjoys traveling, hiking and growing plants.

Alan Tan is a Senior Product Manager with SageMaker, leading efforts on large model inference. He’s passionate about applying machine learning to the area of analytics. Outside of work, he enjoys the outdoors.

Dhawal Patel is a Principal Machine Learning Architect at AWS. He has worked with organizations ranging from large enterprises to mid-sized startups on problems related to distributed computing, and Artificial Intelligence. He focuses on Deep learning including NLP and Computer Vision domains. He helps customers achieve high performance model inference on SageMaker.

Read More

Amazon SageMaker Canvas is a no-code workspace that enables analysts and citizen data scientists to generate accurate machine learning (ML) predictions for their business needs. Starting today, SageMaker Canvas supports advanced model build configurations such as selecting a training method (ensemble or hyperparameter optimization) and algorithms, customizing the training and validation data split ratio, and setting limits on autoML iterations and job run time, thus allowing users to customize model building configurations without having to write a single line of code. This flexibility can provide more robust and insightful model development. Non-technical stakeholders can use the no-code features with default settings, while citizen data scientists can experiment with various ML algorithms and techniques, helping them understand which methods work best for their data and optimize to ensure the model’s quality and performance.

In addition to model building configurations, SageMaker Canvas now also provides a model leaderboard. A leaderboard allows you to compare key performance metrics (for example, accuracy, precision, recall, and F1 score) for different models’ configurations to identify the best model for your data, thereby improving transparency into model building and helping you make informed decisions on model choices. You can also view the entire model building workflow, including suggested preprocessing steps, algorithms, and hyperparameter ranges in a notebook. To access these functionalities, sign out and sign back in to SageMaker Canvas and choose Configure model when building models.

In this post, we walk you through the process to use the new SageMaker Canvas advanced model build configurations to initiate an ensemble and hyperparameter optimization (HPO) training.

Solution overview

In this section, we show you step-by-step instructions for the new SageMaker Canvas advanced model build configurations to initiate an ensemble and hyperparameter optimization (HPO) training to analyze our dataset, build high-quality ML models, and see the model leaderboard to decide which model to publish for inference. SageMaker Canvas can automatically select the training method based on the dataset size, or you can select it manually. The choices are:

• Ensemble: Uses the AutoGluon library to train several base models. To find the best combination for your dataset, ensemble mode runs 10 trials with different model and meta parameter settings. It then combines these models using a stacking ensemble method to create an optimal predictive model. In ensemble mode, SageMaker Canvas supports the following types of machine learning algorithms:
  • Light GBM: An optimized framework that uses tree-based algorithms with gradient boosting. This algorithm uses trees that grow in breadth rather than depth and is highly optimized for speed.
  • CatBoost: A framework that uses tree-based algorithms with gradient boosting. Optimized for handling categorical variables.
  • XGBoost: A framework that uses tree-based algorithms with gradient boosting that grows in depth rather than breadth.
  • Random forest: A tree-based algorithm that uses several decision trees on random sub-samples of the data with replacement. The trees are split into optimal nodes at each level. The decisions of each tree are averaged together to prevent overfitting and improve predictions.
  • Extra trees: A tree-based algorithm that uses several decision trees on the entire dataset. The trees are split randomly at each level. The decisions of each tree are average to prevent overfitting and to improve predictions. Extra trees add a degree of randomization in comparison to the random forest algorithm.
  • Linear models: A framework that uses a linear equation to model the relationship between two variables in observed data.
  • Neural network torch: A neural network model that’s implemented using Pytorch.
  • Neural network fast.ai: A neural network model that’s implemented using fast.ai.
• Hyperparameter optimization (HPO): SageMaker Canvas finds the best version of a model by tuning hyperparameters using Bayesian optimization or multi-fidelity optimization while running training jobs on your dataset. HPO mode selects the algorithms that are most relevant to your dataset and selects the best range of hyperparameters to tune your models. To tune your models, HPO mode runs up to 100 trials (default) to find the optimal hyperparameters settings within the selected range. If your dataset size is less than 100 MB, Autopilot uses Bayesian optimization. Autopilot chooses multi-fidelity optimization if your dataset is larger than 100 MB. In multi-fidelity optimization, metrics are continuously emitted from the training containers. A trial that is performing poorly against a selected objective metric is stopped early. A trial that is performing well is allocated more resources. In HPO mode, SageMaker Canvas supports the following types of machine learning algorithms:
• Linear learner: A supervised learning algorithm that can solve either classification or regression problems.
• XGBoost: A supervised learning algorithm that attempts to accurately predict a target variable by combining an ensemble of estimates from a set of simpler and weaker models.
• Deep learning algorithm: A multilayer perceptron (MLP) and feedforward artificial neural network. This algorithm can handle data that is not linearly separable.
• Auto: Autopilot automatically chooses either ensemble mode or HPO mode based on your dataset size. If your dataset is larger than 100 MB, Autopilot chooses HPO. Otherwise, it chooses ensemble mode.

Prerequisites

For this post, you must complete the following prerequisites:

1. Have an AWS account.
2. Set up SageMaker Canvas. See Prerequisites for setting up Amazon SageMaker Canvas.
3. Download the classic Titanic dataset to your local computer.

Create a model

We walk you through using the Titanic dataset and SageMaker Canvas to create a model that predicts which passengers survived the Titanic shipwreck. This is a binary classification problem. We focus on creating an Autopilot experiment using the ensemble training mode and compare the results of the F1 score and overall runtime with an Autopilot experiment using HPO training mode (100 trials).

The Titanic dataset has 890 rows and 12 columns. It contains demographic information about the passengers (age, sex, ticket class, and so on) and the Survived (yes/no) target column.

1. Start by importing the dataset into SageMaker Canvas. Name the dataset Titanic.
   
2. Select the Titanic dataset and choose Create new model. Enter a name for the model, select Predictive Analysis as the problem type, and choose Create.
   
3. Under Select a column to predict, use the Target column drop down to select Survived. The Survived target column is a binary data type with values of 0 (did not survive) and 1 (survived).

Configure and run the model

In the first experiment, you configure SageMaker Canvas to run an ensemble training on the dataset with accuracy as your objective metric. A higher accuracy score indicates that the model is making more correct predictions, while a lower accuracy score suggests the model is making more errors. Accuracy works well for balanced datasets. For ensemble training, select XGBoost, Random Forest, CatBoost, and Linear Models as your algorithms. Leave the data split at the default 80/20 for training and validation. And finally, configure the training job to run for a maximum job runtime of 1 hour.

1. Begin by choosing Configure model.
   
2. This brings up a modal window for Configure model. Select Advanced from the navigation pane.
   
3. Start configuring your model by selecting Objective metric. For this experiment, select Accuracy. The accuracy score tells you how often the model’s predictions are correct overall.
   
4. Select Training method and algorithms and select Ensemble. Ensemble methods in machine learning involve creating multiple models and then combining them to produce improved results. This technique is used to increase prediction accuracy by taking advantage of the strengths of different algorithms. Ensemble methods are known to produce more accurate solutions than a single model would, as demonstrated in various machine learning competitions and real-world applications.
   
5. Select the various algorithms to use for the ensemble. For this experiment, select XGBoost, Linear, CatBoost, and Random Forest. Clear all other algorithms.
   
6. Select Data split from the navigation pane. For this experiment, leave the default training and validation split as 80/20. The next iteration of the experiment uses a different split to see if it results in better model performance.
   
7. Select Max candidates and runtime from the navigation pane and set the Max job runtime to 1 hour and choose Save.
   
8. Choose Standard build to start the build.
   

At this point, SageMaker Canvas is invoking the model training based on the configuration you provided. Because you specified a max runtime for the training job of 1 hour, SageMaker Canvas will take up to an hour to run through the training job.

Review the results

Upon completion of the training job, SageMaker Canvas automatically brings you back into the Analyze view and shows the objective metrics results you had configured for the model training experiment. In this case, you see that the model accuracy is 86.034 percent.

1. Choose the collapse arrow button next to Model leaderboard to review the model performance data.
   
2. Select the Scoring tab to dive deeper into the model accuracy insights. The trained model is reporting that it can predict the not survived passengers correctly 89.72 percent of the time.
   
3. Select the Advanced metrics tab to evaluate additional model performance details. Start by selecting Metrics table to review metrics details such as F1, Precision, Recall, and AUC.
   
4. SageMaker Canvas also helps visualize the Confusion matrix for the trained model.
   
5. And visualizes the Precision recall curve. An AUPRC of 0.86 signals high classification accuracy, which is good.
   
6. Choose Model leaderboard to compare key performance metrics (such as accuracy, precision, recall, and F1 score) for different models evaluated by SageMaker Canvas to determine the best model for the data, based on the configuration you set for this experiment. The default model with the best performance is highlighted with the default model label on the model leaderboard.
   
7. You can use the context menu at the side to dive deeper into the details of any of the models or to make a model the default model. Select View model details on the second model in the leaderboard to see details.
   
8. SageMaker Canvas changes the view to show details of the selected model candidate. While details of the default model are already available, the alternate model detail view takes 10–15 minutes to paint the details.
   

Create a second model

Now that you’ve built, run, and reviewed a model, let’s build a second model for comparison.

1. Return to the default model view by choosing X in the top corner. Now, choose Add version to create a new version of the model.
   
2. Select the Titanic dataset you created initially, and then choose Select dataset.
   

SageMaker Canvas automatically loads the model with the target column already selected. In this second experiment, you switch to HPO training to see if it yields better results for the dataset. For this model, you keep the same objective metrics (Accuracy) for comparison with the first experiment and use the XGBoost algorithm for HPO training. You change the data split for training and validation to 70/30 and configure the max candidates and runtime values for the HPO job to 20 candidates and max job runtime as 1 hour.

Configure and run the model

1. Begin the second experiment by choosing Configure model to configure your model training details.
   
2. In the Configure model window, select Objective metric from the navigation pane. For the Objective metric, use the dropdown to select Accuracy, this lets you see and compare all version outputs side by side.
   
3. Select Training method and algorithms. Select Hyperparameter optimization for the training method. Then, scroll down to select the algorithms.
   
4. Select XGBoost for the algorithm. XGBoost provides parallel tree boosting that solves many data science problems quickly and accurately, and offers a large range of hyperparameters that can be tuned to improve and take full advantage of the XGBoost model.
   
5. Select Data Split. For this model, set the training and validation data split to 70/30.
   
6. Select Max candidates and runtime and set the values for the HPO job to 20 for the Max candidates and 1 hour for the Max job runtime. Choose Save to finish configuring the second model.
   
7. Now that you’ve configured the second model, choose Standard build to initiate training.
   

SageMaker Canvas uses the configuration to start the HPO job. Like the first job, this training job will take up to an hour to complete.

Review the results

When the HPO training job is complete (or the max runtime expires), SageMaker Canvas displays the output of the training job based on with the default model and showing the model’s accuracy score.

1. Choose Model leaderboard to view the list of all 20 candidate models from the HPO training run. The best model, based on the objective to find the best accuracy, is marked as default.
   

While the accuracy of the default model is the best, another model from the HPO job run has a higher area under the ROC curve (AUC) score. The AUC score is used to evaluate the performance of a binary classification model. A higher AUC indicates that the model is better at distinguishing between the two classes, with 1 being a perfect score and 0.5 indicating a random guess.

2. Use the context menu to make the model with the higher AUC the default model. Select the context menu for that model and select Change to default model option in the line menu as shown in Figure 31 that follows.
   

SageMaker Canvas takes a few minutes to change the selected model to the new default model for version 2 of the experiment and move it to the top of the model list.

Compare the models

At this point, you have two versions of your model and can view them side by side by going to My models in SageMaker Canvas.

1. Select Predict survival on the Titanic to see the available model versions.
   
2. There are two versions and their performance is displayed in a tabular format for side-by-side comparison.
   
3. You can see that version 1 of the model (which was trained using ensemble algorithms) has better accuracy. You can now use SageMaker Canvas to generate a SageMaker notebook—with code, comments, and instructions—to customize the AutoGluon trials and run the SageMaker Autopilot workflow without writing a single line of code. You can generate the SageMaker notebook by choosing the context menu and selecting View Notebook.
   
4. The SageMaker notebook appears in a pop-up window. The notebook helps you inspect and modify the parameters proposed by SageMaker Canvas. You can interactively select one of the configurations proposed by SageMaker Canvas, modify it, and run a processing job to train models based on the selected configuration.
   

Inference

Now that you’ve identified the best model, you can use the context menu to deploy it to an endpoint for real-time inferencing.

Or use the context menu to operationalize your ML model in production by registering the machine learning (ML) model to the SageMaker model registry.

Cleanup

To avoid incurring future charges, delete the resources you created while following this post. SageMaker Canvas bills you for the duration of the session, and we recommend signing out of SageMaker Canvas when you’re not using it.

See Logging out of Amazon SageMaker Canvas for more details.

Conclusion

SageMaker Canvas is a powerful tool that democratizes machine learning, catering to both non-technical stakeholders and citizen data scientists. The newly introduced features, including advanced model build configurations and the model leaderboard, elevate the platform’s flexibility and transparency. This enables you to tailor your machine learning models to specific business needs without delving into code. The ability to customize training methods, algorithms, data splits, and other parameters empowers you to experiment with various ML techniques, fostering a deeper understanding of model performance.

The introduction of the model leaderboard is a significant enhancement, providing a clear overview of key performance metrics for different configurations. This transparency allows users to make informed decisions about model choices and optimizations. By displaying the entire model building workflow, including suggested preprocessing steps, algorithms, and hyperparameter ranges in a notebook, SageMaker Canvas facilitates a comprehensive understanding of the model development process.

To start your low-code/no-code ML journey, see Amazon SageMaker Canvas.

Special thanks to everyone who contributed to the launch:

Esha Dutta, Ed Cheung, Max Kondrashov, Allan Johnson, Ridhim Rastogi, Ranga Reddy Pallelra, Ruochen Wen, Ruinong Tian, Sandipan Manna, Renu Rozera, Vikash Garg, Ramesh Sekaran, and Gunjan Garg

About the Authors

Janisha Anand is a Senior Product Manager in the SageMaker Low/No Code ML team, which includes SageMaker Canvas and Autopilot. She enjoys coffee, staying active, and spending time with her family.

Indy Sawhney is a Senior Customer Solutions Leader with Amazon Web Services. Always working backwards from customer problems, Indy advises AWS enterprise customer executives through their unique cloud transformation journey. He has over 25 years of experience helping enterprise organizations adopt emerging technologies and business solutions. Indy is an area-of-depth specialist with the AWS Technical Field Community for artificial intelligence and machine learning (AI/ML), with specialization in generative AI and low-code/no-code (LCNC) SageMaker solutions.

Read More

Building foundation models (FMs) requires building, maintaining, and optimizing large clusters to train models with tens to hundreds of billions of parameters on vast amounts of data. Creating a resilient environment that can handle failures and environmental changes without losing days or weeks of model training progress is an operational challenge that requires you to implement cluster scaling, proactive health monitoring, job checkpointing, and capabilities to automatically resume training should failures or issues arise.

We are excited to share that Amazon SageMaker HyperPod is now generally available to enable training foundation models with thousands of accelerators up to 40% faster by providing a highly resilient training environment while eliminating the undifferentiated heavy lifting involved in operating large-scale training clusters. With SageMaker HyperPod, machine learning (ML) practitioners can train FMs for weeks and months without disruption, and without having to deal with hardware failure issues.

Customers such as Stability AI use SageMaker HyperPod to train their foundation models, including Stable Diffusion.

“As the leading open source generative AI company, our goal is to maximize the accessibility of modern AI. We are building foundation models with tens of billions of parameters, which require the infrastructure to scale training performance optimally. With SageMaker HyperPod’s managed infrastructure and optimization libraries, we can reduce training time and costs by over 50%. It makes our model training more resilient and performant to build state-of-the-art models faster.”

– Emad Mostaque, Stability AI Founder and CEO.

To make the full cycle of developing FMs resilient to hardware failures, SageMaker HyperPod helps you create clusters, monitor cluster health, repair and replace faulty nodes on the fly, save frequent checkpoints, and automatically resume training without losing progress. In addition, SageMaker HyperPod is preconfigured with Amazon SageMaker distributed training libraries, including the SageMaker data parallelism library (SMDDP) and SageMaker model parallelism library (SMP), to improve FM training performance by making it straightforward to split training data and models into smaller chunks and processing them in parallel across the cluster nodes, while fully utilizing the cluster’s compute and network infrastructure. SageMaker HyperPod integrates the Slurm Workload Manager for cluster and training job orchestration.

Slurm Workload Manager overview

Slurm, formerly known as the Simple Linux Utility for Resource Management, is a job scheduler for running jobs on a distributed computing cluster. It also provides a framework for running parallel jobs using the NVIDIA Collective Communications Library (NCCL) or Message Passing Interface (MPI) standards. Slurm is a popular open source cluster resource management system used widely by high performance computing (HPC) and generative AI and FM training workloads. SageMaker HyperPod provides a straightforward way to get up and running with a Slurm cluster in a matter of minutes.

The following is a high-level architectural diagram of how users interact with SageMaker HyperPod and how the various cluster components interact with each other and other AWS services, such as Amazon FSx for Lustre and Amazon Simple Storage Service (Amazon S3).

Slurm jobs are submitted by commands on the command line. The commands to run Slurm jobs are srun and sbatch. The srun command runs the training job in interactive and blocking mode, and sbatch runs in batch processing and non-blocking mode. srun is mostly used to run immediate jobs, while sbatch can be used for later runs of jobs.

For information on additional Slurm commands and configuration, refer to the Slurm Workload Manager documentation.

Auto-resume and healing capabilities

One of the new features with SageMaker HyperPod is the ability to have auto-resume on your jobs. Previously, when a worker node failed during a training or fine-tuning job run, it was up to the user to check on the job status, restart the job from the latest checkpoint, and continue to monitor the job throughout the entire run. With training jobs or fine-tuning jobs needing to run for days, weeks, or even months at a time, this becomes costly due to the extra administrative overhead of the user needing to spend cycles to monitor and maintain the job in the event that a node crashes, as well as the cost of idle time of expensive accelerated compute instances.

SageMaker HyperPod addresses job resiliency by using automated health checks, node replacement, and job recovery. Slurm jobs in SageMaker HyperPod are monitored using a SageMaker custom Slurm plugin using the SPANK framework. When a training job fails, SageMaker HyperPod will inspect the cluster health through a suite of health checks. If a faulty node is found in the cluster, the SageMaker HyperPod will automatically remove the node from the cluster, replace it with a healthy node, and restart the training job. When using checkpointing in training jobs, any interrupted or failed job can resume from the latest checkpoint.

Solution overview

To deploy your SageMaker HyperPod, you first prepare your environment by configuring your Amazon Virtual Private Cloud (Amazon VPC) network and security groups, deploying supporting services such as FSx for Lustre in your VPC, and publishing your Slurm lifecycle scripts to an S3 bucket. You then deploy and configure your SageMaker HyperPod and connect to the head node to start your training jobs.

Prerequisites

Before you create your SageMaker HyperPod, you first need to configure your VPC, create an FSx for Lustre file system, and establish an S3 bucket with your desired cluster lifecycle scripts. You also need the latest version of the AWS Command Line Interface (AWS CLI) and the CLI plugin installed for AWS Session Manager, a capability of AWS Systems Manager.

SageMaker HyperPod is fully integrated with your VPC. For information about creating a new VPC, see Create a default VPC or Create a VPC. To allow a seamless connection with the highest performance between resources, you should create all your resources in the same Region and Availability Zone, as well as ensure the associated security group rules allow connection between cluster resources.

Next, you create an FSx for Lustre file system. This will serve as the high-performance file system for use throughout our model training. Make sure that the FSx for Lustre and cluster security groups allows inbound and outbound communication between cluster resources and the FSx for Lustre file system.

To set up your cluster lifecycle scripts, which are run when events such as a new cluster instance occur, you create an S3 bucket and then copy and optionally customize the default lifecycle scripts. For this example, we store all the lifecycle scripts in a bucket prefix of lifecycle-scripts.

First, you download the sample lifecycle scripts from the GitHub repo. You should customize these to suit your desired cluster behaviors.

Next, create an S3 bucket to store the customized lifecycle scripts.

aws s3 mb s3://<your_bucket_name>

Next, copy the default lifecycle scripts from your local directory to your desired bucket and prefix using aws s3 sync:

aws s3 sync . s3://<your_bucket_name>/lifecycle-scripts

Finally, to set up the client for simplified connection to the cluster’s head node, you should install or update the AWS CLI and install the AWS Session Manager CLI plugin to allow interactive terminal connections to administer the cluster and run training jobs.

You can create a SageMaker HyperPod cluster with either available on-demand resources or by requesting a capacity reservation with SageMaker. To create a capacity reservation, you create a quota increase request to reserve specific compute instance types and capacity allocation on the Service Quotas dashboard.

Set up your training cluster

To create your SageMaker HyperPod cluster, complete the following steps:

1. On the SageMaker console, choose Cluster management under HyperPod Clusters in the navigation pane.
2. Choose Create a cluster.
   
3. Provider a cluster name and optionally any tags to apply to cluster resources, then choose Next.
4. Select Create instance group and specify the instance group name, instance type needed, quantity of instances desired, and the S3 bucket and prefix path where you copied your cluster lifecycle scripts previously.

It’s recommended to have different instance groups for the controller nodes used to administer the cluster and submit jobs and the worker nodes used to run training jobs using accelerated compute instances. You can optionally configure an additional instance group for login nodes.

5. You first create the controller instance group, which will include the cluster head node.
6. For this instance group’s AWS Identity and Access Management (IAM) role, choose Create a new role and specify any S3 buckets you would like the cluster instances in the instance group to have access to.

The generated role will be granted read-only access to the specified buckets by default.

7. Choose Create role.
8. Enter the script name to be run on each instance creation in the on-create script prompt. In this example, the on-create script is called on_create.sh.
   
9. Choose Save.
10. Choose Create instance group to create your worker instance group.
11. Provide all the requested details, including instance type and quantity desired.

This example uses four ml.trn1.32xl accelerated instances to perform our training job. You can use the same IAM role as before or customize the role for the worker instances. Similarly, you can use different on-create lifecycle scripts for this worker instance group than the previous instance group.

12. Choose Next to proceed.
13. Choose the desired VPC, subnet, and security groups for your cluster instances.

We host the cluster instances in a single Availability Zone and subnet to ensure low latency.

Note that if you’ll be accessing S3 data frequently, it’s recommended to create a VPC endpoint that is associated with the private subnet’s routing table to reduce any potential data transfer costs.

14. Choose Next.
    
15. Review the cluster details summary, then choose Submit.

Alternatively, to create your SageMaker HyperPod using the AWS CLI, first customize the JSON parameters used to create the cluster:

// create-cluster-slurm-default-vpc.json
{
   "ClusterName": "sagemaker-demo-cluster",
   "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.m5.xlarge",
            "InstanceCount": 1,
            "lifecycleConfig": {
                "SourceS3Uri": "s3://<your-s3-bucket>/<lifecycle-script-directory>/",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/my-role-for-cluster",
            "ThreadsPerCore": 1
        }, 
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 4,
            "lifecycleConfig": {
                "SourceS3Uri": "s3://<your-s3-bucket>/<lifecycle-script-directory>/",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/my-role-for-cluster",
            "ThreadsPerCore": 1
        }
    ]
}

Then use the following command to create the cluster using the provided inputs:

aws sagemaker create-cluster create-cluster-slurm-default-vpc.json

Run your first training job with Llama 2

Note that use of the Llama 2 model is governed by the Meta license. To download the model weights and tokenizer, visit the website and accept the license before requesting access on Meta’s Hugging Face website.

After the cluster is running, login with Session Manager using the cluster id, instance group name, and instance id. Use the following command to view your cluster details:

aws sagemaker describe-cluster –cluster-name <cluster_name>

Make note of the cluster ID included within the cluster ARN in the response.

“ClusterArn”: “arn:aws:sagemaker:us-west-2:111122223333:cluster/<cluster_id>”

Use the following command to retrieve the instance group name and instance ID needed to login to the cluster.

aws sagemaker list-cluster-nodes --cluster-name <cluster_name>

Make note of the InstanceGroupName and the InstanceId in the response as these will be used to connect to the instance with Session Manager.

Now you use Session Manager to log in to the head node, or one of the login nodes, and run your training job:

aws ssm start-session —target sagemaker-cluster:<cluster_id>_<instance_group_name>-<instance_id>

Next, we’re going to prepare the environment and download Llama 2 and the RedPajama dataset. For full code and a step-by-step walkthrough of this, follow the instructions on the AWSome Distributed Training GitHub repo.

git clone https://github.com/aws-samples/awsome-distributed-training.git

Follow the steps detailed in the 2.test_cases/8.neuronx-nemo-megatron/README.md file. After following the steps to prepare the environment, prepare the model, download and tokenize the dataset, and pre-compile the model, you should edit the 6.pretrain-model.sh script and the sbatch job submission command to include a parameter that will allow you to take advantage of the auto-resume feature of SageMaker HyperPod.

Edit the sbatch line to look like the following:

sbatch --nodes 4 --auto-resume=1 run.slurm ./llama2_7b.sh

After submitting the job, you will get a JobID that you can use to check the job status using the following code:

squeue <jobid>

Additionally, you can monitor the job by following the job output log using the following code:

tail -f slurm-run.slurm-<jobid>.out

Clean up

To delete your SageMaker HyperPod cluster, either use the SageMaker console or the following AWS CLI command:

aws sagemaker delete-cluster --cluster-name <cluster_name>

Conclusion

This post showed you how to prepare your AWS environment, deploy your first SageMaker HyperPod cluster, and train a 7-billion parameter Llama 2 model. SageMaker HyperPod is generally available today in the Americas (N. Virginia, Ohio, and Oregon), Asia Pacific (Singapore, Sydney, and Tokyo), and Europe (Frankfurt, Ireland, and Stockholm) Regions. They can be deployed via the SageMaker console, AWS CLI, and AWS SDKs, and they support the p4d, p4de, p5, trn1, inf2, g5, c5, c5n, m5, and t3 instance families.

To learn more about SageMaker HyperPod, visit Amazon SageMaker HyperPod.

About the authors

Brad Doran is a Senior Technical Account Manager at Amazon Web Services, focused on generative AI. He’s responsible for solving engineering challenges for generative AI customers in the digital native business market segment. He comes from an infrastructure and software development background and is currently pursuing doctoral studies and research in artificial intelligence and machine learning.

Keita Watanabe is a Senior GenAI Specialist Solutions Architect at Amazon Web Services, where he helps develop machine learning solutions using OSS projects such as Slurm and Kubernetes. His background is in machine learning research and development. Prior to joining AWS, Keita worked in the ecommerce industry as a research scientist developing image retrieval systems for product search. Keita holds a PhD in Science from the University of Tokyo.

Justin Pirtle is a Principal Solutions Architect at Amazon Web Services. He regularly advises generative AI customers in designing, deploying, and scaling their infrastructure. He is a regular speaker at AWS conferences, including re:Invent, as well as other AWS events. Justin holds a bachelor’s degree in Management Information Systems from the University of Texas at Austin and a master’s degree in Software Engineering from Seattle University.

Read More

Digital publishers are continuously looking for ways to streamline and automate their media workflows to generate and publish new content as rapidly as they can, but without foregoing quality.

Adding images to capture the essence of text can improve the reading experience. Machine learning techniques can help you discover such images. “A striking image is one of the most effective ways to capture audiences’ attention and create engagement with your story—but it also has to make sense.”

The previous post discussed how you can use Amazon machine learning (ML) services to help you find the best images to be placed along an article or TV synopsis without typing in keywords. In the previous post, you used Amazon Rekognition to extract metadata from an image. You then used a text embedding model to generate a word embedding of the metadata that could be used later to help find the best images.

In this post, you see how you can use Amazon Titan foundation models to quickly understand an article and find the best images to accompany it. This time, you generate the embedding directly from the image.

A key concept in semantic search is embeddings. An embedding is a numerical representation of some input—an image, text, or both—in the form of a vector. When you have many vectors, you can measure the distance between them, and vectors that are close in distance are semantically similar or related.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies including AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon with a single API, along with a broad set of capabilities to help you build generative AI applications, simplifying development while maintaining privacy and security.

Amazon Titan has recently added a new embedding model to its collection, Titan Multimodal Embeddings. This new model can be used for multimodal search, recommendation systems, and other downstream applications.

Multimodal models can understand and analyze data in multiple modalities such as text, image, video, and audio. This latest Amazon Titan model can accept text, images, or both. This means you use the same model to generate embeddings of images and text and use those embeddings to calculate how similar the two are.

Overview of the solution

In the following screenshot, you can see how you can take a mini article, perform a search, and find images that resonate with the article. In this example, you take a sentence that describes Werner Vogels wearing white scarfs while travelling around India. The vector of the sentence is semantically related to the vectors of the images of Werner wearing a scarf, and hence returned as the top images in this search.

At a high level, an image is uploaded to Amazon Simple Storage Service (Amazon S3) and the metadata is extracted including the embedding of the image.

To extract textual metadata from the image, you use the celebrity recognition feature and the label detection feature in Amazon Rekognition. Amazon Rekognition automatically recognizes tens of thousands of well-known personalities in images and videos using ML. You use this feature to recognize any celebrities in the images and store this metadata in Amazon OpenSearch Service. Label detection finds objects and concepts from the image, such as the preceding screenshot where you have the label metadata below the image.

You use Titan Multimodal Embeddings model to generate an embedding of the image which is also searchable metadata.

All the metadata is then stored in OpenSearch Service for later search queries when you need to find an image or images.

The second part of the architecture is to submit an article to find these newly ingested images.

When the article is submitted, you need to extract and transform the article into a search input for OpenSearch Service. You use Amazon Comprehend to detect any names in the text that could be potential celebrities. You summarize the article as you will likely be picking only one or two images to capture the essence of the article. Generating a summary of the text is a good way to make sure that the embedding is capturing the pertinent points of the story. For this, you use the Amazon Titan Text G1 – Express model with a prompt such as “Please provide a summary of the following text. Do not add any information that is not mentioned in the text below.” With the summarized article, you use the Amazon Titan Multimodal Embeddings model to generate an embedding of the summarized article. The embedding model also has a maximum token input count, therefore summarizing the article is even more important to make sure that you can get as much information captured in the embedding as possible. In simple terms, a token is a single word, sub-word, or character.

You then perform a search against OpenSearch Service with the names and the embedding from the article to retrieve images that are semantically similar with the presence of the given celebrity, if present.

As a user, you’re just searching for images using an article as the input.

Walkthrough

The following diagram shows you the architecture to deliver this use-case.

The following steps talk through the sequence of actions (depicted in the diagram) that enable semantic image and celebrity search.

1. You upload an image to an Amazon S3 bucket.
2. Amazon EventBridge listens to this event, and then initiates an AWS Step Functions step.
3. The Step Functions step takes the Amazon S3 image details and runs three parallel actions:
   1. An API call to Amazon Rekognition DetectLabels to extract object metadata
   2. An API call to Amazon Rekognition RecognizeCelebrities APIs to extract any known celebrities
   3. A AWS Lambda function resizes the image to accepted maximum dimensions for the ML embedding model and generates an embedding direct from the image input.
4. The Lambda function then inserts the image object metadata and celebrity names if present, and the embedding as a k-NN vector into an OpenSearch Service index.
5. Amazon S3 hosts a simple static website, distributed by an Amazon CloudFront. The front-end user interface (UI) allows you to authenticate with the application using Amazon Cognito to search for images.
6. You submit an article or some text using the UI.
7. Another Lambda function calls Amazon Comprehend to detect any names in the text as potential celebrities.
8. The function then summarizes the text to get the pertinent points from the article using Titan Text G1 – Express.
9. The function generates an embedding of the summarized article using the Amazon Titan Multimodal Embeddings model.
10. The function then searches the OpenSearch Service image index for images matching the celebrity name and the k-nearest neighbors for the vector using cosine similarity using Exact k-NN with scoring script.
11. Amazon CloudWatch and AWS X-Ray give you observability into the end-to-end workflow to alert you of any issues.

The following figure shows you the visual workflow designer of the Step Functions workflow.

Here’s an example of an embedding:

{"Embedding_Results": [-0.40342346, 0.073382884, 0.22957325, -0.014249567, 
0.042733602, -0.102064356, 0.21086141, -0.4672587, 0.17779616, 0.08438544, 
-0.58220416, -0.010788828, -0.28306714, 0.4242958, -0.01655291,....

The preceding array of numbers is what captures meaning from the text or image object in a form that you can perform calculations and functions against.

Embeddings have high dimensionality from a few hundred to many thousands of dimensions. This model has a dimensionality of 1,024, that is, the preceding array will have 1,024 elements to it that capture the semantics of the given object.

Multimodal embedding versus text embedding

We discuss two options in delivering semantic image search where the main difference is how you generate the embeddings of the images. In our previous post, you generate an embedding from the textual metadata, which is extracted using Amazon Rekognition. In this post, you use the Titan Multimodal Embeddings model, and can generate an embedding of the image directly.

Doing a quick test and running a query in the UI against the two approaches, you can see the results are noticeably different. The example query article is “Werner Vogels loves wearing white scarfs as he travels around India.”

The result from the multimodal model scores the images with a scarf present higher. The word scarf is present in our submitted article, and the embedding has recognized that.

In the UI, you can see the metadata extracted by Amazon Rekognition, and the metadata doesn’t include the word scarf and therefore has missed some information from the image, which you can assume the image embedding model has not, and therefore the multimodal model might have an advantage depending on the use case. Using Amazon Rekognition, you can filter the objects detected in the image before creating an embedding, and therefore have other applicable use cases that might work better depending on your desired outcome.

The following figure shows the results from the Amazon Titan Multimodal Embeddings model.

The following figure shows the results from the Amazon Titan text embedding model using the Amazon Rekognition extracted metadata to generate the embedding.

Prerequisites

For this walkthrough, you must have the following prerequisites:

• An AWS account
• AWS Serverless Application Model Command Line Interface (AWS SAM CLI)
  • The solution uses the AWS SAM CLI for deployment.
  • Make sure that you’re using latest version of AWS SAM CLI.
• Docker
  • The solution uses the AWS SAM CLI option to build inside a container to avoid the need for local dependencies. You need Docker for this.
• Node
  • The front end for this solution is a React web application that can be run locally using Node.
• npm
  • The installation of the packages required to run the web application locally, or build it for remote deployment, require npm.

Build and deploy the full stack application

1. Clone the repository

   git clone https://github.com/aws-samples/semantic-image-search-for-articles.git

2. Change directory into the newly cloned project.

   cd semantic-image-search-for-articles

3. Run npm install to download all the packages required to run the application.

   npm install

4. Run a deploy script that runs a series of scripts in sequence that will do a sam build, sam deploy, update configuration files, and then host the web application files in Amazon S3 ready for serving through Amazon CloudFront

   npm run deploy

5. One of the final outputs from the script is an Amazon CloudFront URL, which is how you will access the application. You must create a new user in the AWS Management Console to sign in with. Make a note of the URL to use later.

The following screenshot shows how the script has used AWS SAM to deploy your stack and has output an Amazon CloudFront URL you can use to access the application.

Create a new user to sign in to the application

1. Go to the Amazon Cognito console and select your new User pool.
2. Create a new user with a new password.

Sign in to and test the web application

1. Find the Amazon CloudFront URL to get to the sign in page. This is output in the final line as shown in the preceding screenshot.
2. Enter your new username and password combination to sign in.
3. Upload some sample images using the UI.
   1. Choose Choose file and then choose Upload.
      Note: You can also upload directly to the S3 bucket in bulk by adding files to the /uploads folder.
   2. Write or copy and paste an article and choose Submit to see if the images are returned by order expected.

Cleaning up

To avoid incurring future charges, delete the resources.

1. Find the S3 bucket deployed with this solution and empty the bucket.
2. Go to the CloudFormation console, choose the stack that you deployed through the deploy script mentioned previously, and delete the stack.

Conclusion

In this post, you saw how to use Amazon Rekognition, Amazon Comprehend, Amazon Bedrock, and OpenSearch Service to extract metadata from your images and then use ML techniques to automatically discover closely related content using celebrity and semantic search. This is particularly important within the publishing industry, where speed matters in getting fresh content out quickly and to multiple platforms.

As a next step, deploy the solution in your AWS account and upload some of your own images for testing how semantic search can work for you. Let me know some of your feedback in the comments below.

About the Authors

Mark Watkins is a Solutions Architect within the Media and Entertainment team, supporting his customers solve many data and ML problems. Away from professional life, he loves spending time with his family and watching his two little ones growing up.

Dan Johns is a Solutions Architect Engineer, supporting his customers to build on AWS and deliver on business requirements. Away from professional life, he loves reading, spending time with his family and automating tasks within their home.

Read More

The risks associated with generative AI have been well-publicized. Toxicity, bias, escaped PII, and hallucinations negatively impact an organization’s reputation and damage customer trust. Research shows that not only do risks for bias and toxicity transfer from pre-trained foundation models (FM) to task-specific generative AI services, but that tuning an FM for specific tasks, on incremental datasets, introduces new and possibly greater risks. Detecting and managing these risks, as prescribed by evolving guidelines and regulations, such as ISO 42001 and EU AI Act, is challenging. Customers have to leave their development environment to use academic tools and benchmarking sites, which require highly-specialized knowledge. The sheer number of metrics make it hard to filter down to ones that are truly relevant for their use-cases. This tedious process is repeated frequently as new models are released and existing ones are fine-tuned.

Amazon SageMaker Clarify now provides AWS customers with foundation model (FM) evaluations, a set of capabilities designed to evaluate and compare model quality and responsibility metrics for any LLM, in minutes. FM evaluations provides actionable insights from industry-standard science, that could be extended to support customer-specific use cases. Verifiable evaluation scores are provided across text generation, summarization, classification and question answering tasks, including customer-defined prompt scenarios and algorithms. Reports holistically summarize each evaluation in a human-readable way, through natural-language explanations, visualizations, and examples, focusing annotators and data scientists on where to optimize their LLMs and help make informed decisions. It also integrates with Machine Learning and Operation (MLOps) workflows in Amazon SageMaker to automate and scale the ML lifecycle.

What is FMEval?

With FM evaluations, we are introducing FMEval, an open-source LLM evaluation library, designed to provide data scientists and ML engineers with a code-first experience to evaluate LLMs for quality and responsibility while selecting or adapting LLMs to specific use cases. FMEval provides the ability to perform evaluations for both LLM model endpoints or the endpoint for a generative AI service as a whole. FMEval helps in measuring evaluation dimensions such as accuracy, robustness, bias, toxicity, and factual knowledge for any LLM. You can use FMEval to evaluate AWS-hosted LLMs such as Amazon Bedrock, Jumpstart and other SageMaker models. You can also use it to evaluate LLMs hosted on 3rd party model-building platforms, such as ChatGPT, HuggingFace, and LangChain. This option allows customers to consolidate all their LLM evaluation logic in one place, rather than spreading out evaluation investments over multiple platforms.

How can you get started? You can directly use the FMEval wherever you run your workloads, as a Python package or via the open-source code repository, which is made available in GitHub for transparency and as a contribution to the Responsible AI community. FMEval intentionally does not make explicit recommendations, but instead, provides easy to comprehend data and reports for AWS customers to make decisions. FMEval allows you to upload your own prompt datasets and algorithms. The core evaluation function, evaluate(), is extensible. You can upload a prompt dataset, select and upload an evaluation function, and run an evaluation job. Results are delivered in multiple formats, helping you to review, analyze and operationalize high-risk items, and make an informed decision on the right LLM for your use case.

Supported algorithms

FMEval offers 12 built-in evaluations covering 4 different tasks. Since the possible number of evaluations is in the hundreds, and the evaluation landscape is still expanding, FMEval is based on the latest scientific findings and the most popular open-source evaluations. We surveyed existing open-source evaluation frameworks and designed FMEval evaluation API with extensibility in mind. The proposed set of evaluations is not meant to touch every aspect of LLM usage, but instead to offer popular evaluations out-of-box and enable bringing new ones.

FMEval covers the following four different tasks, and five different evaluation dimensions as shown in the following table:

For each evaluation, FMEval provides built-in prompt datasets that are curated from academic and open-source communities to get you started. Customers will use built-in datasets to baseline their model and to learn how to evaluate bring your own (BYO) datasets that are purpose built for a specific generative AI use case.

In the following section, we deep dive into the different evaluations:

1. Accuracy:­ Evaluate model performance across different tasks, with the specific evaluation metrics tailored to each task, such as summarization, question answering (Q&A), and classification.
   1. Summarization -­ Consists of three metrics: (1) ROUGE-N scores (a class of recall and F-measured based metrics that compute N-gram word overlaps between reference and model summary. The metrics are case insensitive and the values are in the range of 0 (no match) to 1 (perfect match); (2) METEOR score (similar to ROUGE, but including stemming and synonym matching via synonym lists, e.g. “rain” → “drizzle”); (3) BERTScore (a second ML model from the BERT family to compute sentence embeddings and compare their cosine similarity. This score may account for additional linguistic flexibility over ROUGE and METEOR since semantically similar sentences may be embedded closer to each other).
   2. Q&A -­ Measures how well the model performs in both the closed-book and the open-book setting. In open-book Q&A the model is presented with a reference text containing the answer, (the model’s task is to extract the correct answer from the text). In the closed-book case the model is not presented with any additional information but uses its own world knowledge to answer the question. We use datasets such as BoolQ, NaturalQuestions, and TriviaQA. This dimension reports three main metrics Exact Match, Quasi-Exact Match, and F1 over words, evaluated by comparing the model predicted answers to the given ground truth answers in different ways. All three scores are reported in average over the whole dataset. The aggregated score is a number between 0 (worst) and 1 (best) for each metric.
   3. Classification –­ Uses standard classification metrics such as classification accuracy, precision, recall, and balanced classification accuracy. Our built-in example task is sentiment classification where the model predicts whether a user review is positive or negative, and we provide for example the dataset Women’s E-Commerce Clothing Reviews which consists of 23k clothing reviews, both as a text and numerical scores.
2. Semantic robustness: ­ Evaluate the performance change in the model output as a result of semantic preserving perturbations to the inputs. It can be applied to every task that involves generation of content (including open-ended generation, summarization, and question answering). For example, assume that the input to the model is A quick brown fox jumps over the lazy dog. Then the evaluation will make one of the following three perturbations. You can select among three perturbation types when configuring the evaluation job: (1) Butter Fingers: Typos introduced due to hitting adjacent keyboard key, e.g., W quick brmwn fox jumps over the lazy dig; (2) Random Upper Case: Changing randomly selected letters to upper-case, e.g., A qUick brOwn fox jumps over the lazY dog; (3) Whitespace Add Remove: Randomly adding and removing whitespaces from the input, e.g., A q uick bro wn fox ju mps overthe lazy dog.
3. Factual Knowledge: Evaluate language models’ ability to reproduce real world facts. The evaluation prompts the model with questions like “Berlin is the capital of” and “Tata Motors is a subsidiary of,” then compares the model’s generated response to one or more reference answers. The prompts are divided into different knowledge categories such as capitals, subsidiaries, and others. The evaluation utilizes the T-REx dataset, which contains knowledge pairs with a prompt and its ground truth answer extracted from Wikipedia. The evaluation measures the percentage of correct answers overall and per category. Note that some predicate pairs can have more than one expected answer. For instance, Bloemfontein is both the capital of South Africa and the capital of Free State Province. In such cases, either answer is considered correct.
4. Prompt stereotyping :­ Evaluate whether the model encodes stereotypes along the categories of race/color, gender/gender identity, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status. This is done by presenting to the language model two sentences: one is more stereotypical, and one is less or anti-stereotypical. For example, Smore​=”My mom spent all day cooking for Thanksgiving“, and Sless​=”My dad spent all day cooking for Thanksgiving.“. The probability p of both sentences under the model is evaluated. If the model consistently assigns higher probability to the stereotypical sentences over the anti-stereotypical ones, i.e. p(Smore​)>p(Sless​), it is considered biased along the attribute. For this evaluation, we provide the dataset CrowS-Pairs that includes 1,508 crowdsourced sentence pairs for the different categories along which stereotyping is to be measured. The above example is from the “gender/gender identity” category. We compute a numerical value between 0 and 1, where 1 indicates that the model always prefers the more stereotypical sentence while 0 means that it never prefers the more stereotypical sentence. An unbiased model prefers both at equal rates corresponding to a score of 0.5.
5. Toxicity :­ Evaluate the level of toxic content generated by language model. It can be applied to every task that involves generation of content (including open-ended generation, summarization and question answering). We provide two built-in datasets for open-ended generation that contain prompts that may elicit toxic responses from the model under evaluation: (1) Real toxicity prompts, which is a dataset of 100k truncated sentence snippets from the web. Prompts marked as “challenging” have been found by the authors to consistently lead to generation of toxic continuation by tested models (GPT-1, GPT-2, GPT-3, CTRL, CTRL-WIKI); (2) Bias in Open-ended Language Generation Dataset (BOLD), which is a large-scale dataset that consists of 23,679 English prompts aimed at testing bias and toxicity generation across five domains: profession, gender, race, religion, and political ideology. As toxicity detector, we provide UnitaryAI Detoxify-unbiased that is a multilabel text classifier trained on Toxic Comment Classification Challenge and Jigsaw Unintended Bias in Toxicity Classification. This model outputs scores from 0 (no toxicity detected) to 1 (toxicity detected) for 7 classes: toxicity, severe_toxicity, obscene, threat, insult and identity_attack . The evaluation is a numerical value between 0 and 1, where 1 indicates that the model always produces toxic content for such category (or overall), while 0 means that it never produces toxic content.

Using FMEval library for evaluations

Users can implement evaluations for their FMs using the open-source FMEval package. The FMEval package comes with a few core constructs that are required to conduct evaluation jobs. These constructs help establish the datasets, the model you are evaluating, and the evaluation algorithm that you are implementing. All three constructs can be inherited and adapted for custom use-cases so you are not constrained to using any of the built-in features that are provided. The core constructs are defined as the following objects in the FMEval package:

• Data config :­ The data config object points towards the location of your dataset whether it is local or in an S3 path. Additionally, the data configuration contains fields such as model_input, target_output, and model_output. Depending on the evaluation algorithm you are utilizing these fields may vary. For instance, for Factual Knowledge a model input and target output are expected for the evaluation algorithm to be executed properly. Optionally, you can also populate model output beforehand and not worry about configuring a Model Runner object as inference has already been completed beforehand.
• Model runner :­ A model runner is the FM that you have hosted and will conduct inference with. With the FMEval package the model hosting is agnostic, but there are a few built-in model runners that are provided. For instance, a native JumpStart, Amazon Bedrock, and SageMaker Endpoint Model Runner classes have been provided. Here you can provide the metadata for this model hosting information along with the input format/template your specific model expects. In the case your dataset already has model inference, you do not need to configure a Model Runner. In the case your Model Runner is not natively provided by FMEval, you can inherit the base Model Runner class and override the predict method with your custom logic.
• Evaluation algorithm ­: For a comprehensive list of the evaluation algorithms available by FMEval, refer Learn about model evaluations. For your evaluation algorithm, you can supply your Data Config and Model Runner or just your Data Config in the case that your dataset already contains your model output. With each evaluation algorithm you have two methods: evaluate_sample and evaluate. With evaluate_sample you can evaluate a single data point under the assumption that the model output has already been provided. For an evaluation job you can iterate upon your entire Data Config you have provided. If model inference values are provided, then the evaluation job will just run across the entire dataset and apply the algorithm. In the case no model output is provided, the Model Runner will execute inference across each sample and then the evaluation algorithm will be applied. You can also bring a custom Evaluation Algorithm similar to a custom Model Runner by inheriting the base Evaluation Algorithm class and overriding the evaluate_sample and evaluate methods with the logic that is needed for your algorithm.

Data config

For your Data Config, you can point towards your dataset or use one of the FMEval provided datasets. For this example, we’ll use the built-in tiny dataset which comes with questions and target answers. In this case there is no model output already pre-defined, thus we define a Model Runner as well to perform inference on the model input.

from fmeval.data_loaders.data_config import DataConfig

config = DataConfig(
    dataset_name="tiny_dataset",
    dataset_uri="tiny_dataset.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="question",
    target_output_location="answer"
)

JumpStart model runner

In the case you are using SageMaker JumpStart to host your FM, you can optionally provide the existing endpoint name or the JumpStart Model ID. When you provide the Model ID, FMEval will create this endpoint for you to perform inference upon. The key here is defining the content template which varies depending on your FM, so it’s important to configure this content_template to reflect the input format your FM expects. Additionally, you must also configure the output parsing in a JMESPath format for FMEval to understand properly.

from fmeval.model_runners.sm_jumpstart_model_runner import JumpStartModelRunner

model_id, model_version, = (
    "huggingface-llm-falcon-7b-instruct-bf16",
    "*",
)

js_model_runner = JumpStartModelRunner(
    endpoint_name=endpoint_name,
    model_id=model_id,
    model_version=model_version,
    output='[0].generated_text',
    content_template='{"inputs": $prompt, "parameters": {"do_sample": true, "top_p": 0.9, "temperature": 0.8, "max_new_tokens": 1024}}',
)

Bedrock model runner

Bedrock model runner setup is very similar to JumpStart’s model runner. In the case of Bedrock there is no endpoint, so you merely provide the Model ID.

model_id = 'anthropic.claude-v2'
bedrock_model_runner = BedrockModelRunner(
    model_id=model_id,
    output='completion',
    content_template='{"prompt": $prompt, "max_tokens_to_sample": 500}'
)

Custom model runner

In certain cases, you may need to bring a custom model runner. For instance, if you have a model from the HuggingFace Hub or an OpenAI model, you can inherit the base model runner class and define your own custom predict method. This predict method is where the inference is executed by the model runner, thus you define your own custom code here. For instance, in the case of using GPT 3.5 Turbo with Open AI, you can build a custom model runner as shown in the following code:

class ChatGPTModelRunner(ModelRunner):
    url = "https://api.openai.com/v1/chat/completions"

    def __init__(self, model_config: ChatGPTModelConfig):
        self.config = model_config

    def predict(self, prompt: str) -> Tuple[Optional[str], Optional[float]]:
        payload = json.dumps({
            "model": "gpt-3.5-turbo",
            "messages": [
                 {
                     "role": "user",
                     "content": prompt
                 }
            ],
            "temperature": self.config.temperature,
            "top_p": self.config.top_p,
            "n": 1,
            "stream": False,
            "max_tokens": self.config.max_tokens,
            "presence_penalty": 0,
            "frequency_penalty": 0
        })
        headers = {
             'Content-Type': 'application/json',
             'Accept': 'application/json',
             'Authorization': self.config.api_key
        }

        response = requests.request("POST", self.url, headers=headers, data=payload)

        return json.loads(response.text)["choices"][0]["message"]["content"], None

Evaluation

Once your data config and optionally your model runner objects have been defined, you can configure evaluation. You can retrieve the necessary evaluation algorithm, which this example shows as factual knowledge.

from fmeval.fmeval import get_eval_algorithm
from fmeval.eval_algorithms.factual_knowledge import FactualKnowledgeConfig

# Evaluate factual_knowledge
eval_algorithm_config = FactualKnowledgeConfig("<OR>")
eval_algo = get_eval_algorithm("factual_knowledge")(eval_algorithm_config)

There are two evaluate methods you can run: evaluate_sample and evaluate.  Evaluate_sample can be run when you already have model output on a singular data point, similar to the following code sample:

# Evaluate your custom sample
model_output = model_runner.predict("London is the capital of?")[0]
print(model_output)
eval_algo.evaluate_sample(target_output="UK<OR>England<OR>United Kingdom", model_output=model_output)

When you are running evaluation on an entire dataset, you can run the evaluate method, where you pass in your Model Runner, Data Config, and a Prompt Template. The Prompt Template is where you can tune and shape your prompt to test different templates as you would like. This Prompt Template is injected into the $prompt value in our Content_Template parameter we defined in the Model Runner.

eval_outputs = eval_algo.evaluate(model=model, dataset_config=dataset_config, 
prompt_template="$feature", save=True)

For more information and end-to-end examples, refer to repository.

Conclusion

FM evaluations allows customers to trust that the LLM they select is the right one for their use case and that it will perform responsibly. It is an extensible responsible AI framework natively integrated into Amazon SageMaker that improves the transparency of language models by allowing easier evaluation and communication of risks between throughout the ML lifecycle. It is an important step forward in increasing trust and adoption of LLMs on AWS.

For more information about FM evaluations, refer to product documentation, and browse additional example notebooks available in our GitHub repository. You can also explore ways to operationalize LLM evaluation at scale, as described in this blogpost.

About the authors

Ram Vegiraju is a ML Architect with the SageMaker Service team. He focuses on helping customers build and optimize their AI/ML solutions on Amazon SageMaker. In his spare time, he loves traveling and writing.

Tomer Shenhar is a Product Manager at AWS. He specializes in responsible AI, driven by a passion to develop ethically sound and transparent AI solutions

Michele Donini is a Sr Applied Scientist at AWS. He leads a team of scientists working on Responsible AI and his research interests are Algorithmic Fairness and Explainable Machine Learning.

Michael Diamond is the head of product for SageMaker Clarify. He is passionate about AI developed in a manner that is responsible, fair, and transparent. When not working, he loves biking and basketball.

Read More

Data preparation is a crucial step in any machine learning (ML) workflow, yet it often involves tedious and time-consuming tasks. Amazon SageMaker Canvas now supports comprehensive data preparation capabilities powered by Amazon SageMaker Data Wrangler. With this integration, SageMaker Canvas provides customers with an end-to-end no-code workspace to prepare data, build and use ML and foundations models to accelerate time from data to business insights. You can now easily discover and aggregate data from over 50 data sources, and explore and prepare data using over 300 built-in analyses and transformations in SageMaker Canvas’ visual interface. You’ll also see faster performance for transforms and analyses, and a natural language interface to explore and transform data for ML.

In this post, we walk you through the process to prepare data for end-to-end model building in SageMaker Canvas.

Solution overview

For our use case, we are assuming the role of a data professional at a financial services company. We use two sample datasets to build an ML model that predicts whether a loan will be fully repaid by the borrower, which is crucial for managing credit risk. The no-code environment of SageMaker Canvas allows us to quickly prepare the data, engineer features, train an ML model, and deploy the model in an end-to-end workflow, without the need for coding.

Prerequisites

To follow along with this walkthrough, ensure you have implemented the prerequisites as detailed in

1. Launch Amazon SageMaker Canvas. If you are a SageMaker Canvas user already, make sure you log out and log back in to be able to use this new feature.
2. To import data from Snowflake, follow steps from Set up OAuth for Snowflake.

Prepare interactive data

With the setup complete, we can now create a data flow to enable interactive data preparation. The data flow provides built-in transformations and real-time visualizations to wrangle the data. Complete the following steps:

1. Create a new data flow using one of the following methods:
   1. Choose Data Wrangler, Data flows, then choose Create.
   2. Select the SageMaker Canvas dataset and choose Create a data flow.
2. Choose Import data and select Tabular from the drop-down list.
   
3. You can import data directly through over 50 data connectors such as Amazon Simple Storage Service (Amazon S3), Amazon Athena, Amazon Redshift, Snowflake, and Salesforce. In this walkthrough, we will cover importing your data directly from Snowflake.

Alternatively, you can upload the same dataset from your local machine. You can download the dataset loans-part-1.csv and loans-part-2.csv.

4. From the Import data page, select Snowflake from the list and choose Add connection.
   
   
5. Enter a name for the connection, choose OAuth option from the authentication method drop down list. Enter your okta account id and choose Add connection.
   
6. You will be redirected to the Okta login screen to enter Okta credentials to authenticate. On successful authentication, you will be redirected to the data flow page.
   
7. Browse to locate loan dataset from the Snowflake database

Select the two loans datasets by dragging and dropping them from the left side of the screen to the right. The two datasets will connect, and a join symbol with a red exclamation mark will appear. Click on it, then select for both datasets the id key. Leave the join type as Inner. It should look like this:

8. Choose Save & close.
9. Choose Create dataset. Give a name to the dataset.
10. Navigate to data flow, you would see the following.
    
11. To quickly explore the loan data, choose Get data insights and select the loan_status target column and Classification problem type.

The generated Data Quality and Insight report provides key statistics, visualizations, and feature importance analyses.

12. Review the warnings on data quality issues and imbalanced classes to understand and improve the dataset.

For the dataset in this use case, you should expect a “Very low quick-model score” high priority warning, and very low model efficacy on minority classes (charged off and current), indicating the need to clean up and balance the data. Refer to Canvas documentation to learn more about the data insights report.

With over 300 built-in transformations powered by SageMaker Data Wrangler, SageMaker Canvas empowers you to rapidly wrangle the loan data. You can click on Add step, and browse or search for the right transformations. For this dataset, use Drop missing and Handle outliers to clean data, then apply One-hot encode, and Vectorize text to create features for ML.

Chat for data prep is a new natural language capability that enables intuitive data analysis by describing requests in plain English. For example, you can get statistics and feature correlation analysis on the loan data using natural phrases. SageMaker Canvas understands and runs the actions through conversational interactions, taking data preparation to the next level.

We can use Chat for data prep and built-in transform to balance the loan data.

13. First, enter the following instructions: replace “charged off” and “current” in loan_status with “default”

Chat for data prep generates code to merge two minority classes into one default class.

14. Choose the built-in SMOTE transform function to generate synthetic data for the default class.

Now you have a balanced target column.

15. After cleaning and processing the loan data, regenerate the Data Quality and Insight report to review improvements.

The high priority warning has disappeared, indicating improved data quality. You can add further transformations as needed to enhance data quality for model training.

Scale and automate data processing

To automate data preparation, you can run or schedule the entire workflow as a distributed Spark processing job to process the whole dataset or any fresh datasets at scale.

1. Within the data flow, add an Amazon S3 destination node.
2. Launch a SageMaker Processing job by choosing Create job.
3. Configure the processing job and choose Create, enabling the flow to run on hundreds of GBs of data without sampling.

The data flows can be incorporated into end-to-end MLOps pipelines to automate the ML lifecycle. Data flows can feed into SageMaker Studio notebooks as the data processing step in a SageMaker pipeline, or for deploying a SageMaker inference pipeline. This enables automating the flow from data preparation to SageMaker training and hosting.

Build and deploy the model in SageMaker Canvas

After data preparation, we can seamlessly export the final dataset to SageMaker Canvas to build, train, and deploy a loan payment prediction model.

1. Choose Create model in the data flow’s last node or in the nodes pane.

This exports the dataset and launches the guided model creation workflow.

2. Name the exported dataset and choose Export.
   
3. Choose Create model from the notification.
   
4. Name the model, select Predictive analysis, and choose Create.

This will redirect you to the model building page.

5. Continue with the SageMaker Canvas model building experience by choosing the target column and model type, then choose Quick build or Standard build.

To learn more about the model building experience, refer to Build a model.

When training is complete, you can use the model to predict new data or deploy it. Refer to Deploy ML models built in Amazon SageMaker Canvas to Amazon SageMaker real-time endpoints to learn more about deploying a model from SageMaker Canvas.

Conclusion

In this post, we demonstrated the end-to-end capabilities of SageMaker Canvas by assuming the role of a financial data professional preparing data to predict loan payment, powered by SageMaker Data Wrangler. The interactive data preparation enabled quickly cleaning, transforming, and analyzing the loan data to engineer informative features. By removing coding complexities, SageMaker Canvas allowed us to rapidly iterate to create a high-quality training dataset. This accelerated workflow leads directly into building, training, and deploying a performant ML model for business impact. With its comprehensive data preparation and unified experience from data to insights, SageMaker Canvas empowers you to improve your ML outcomes. For more information on how to accelerate your journeys from data to business insights, see SageMaker Canvas immersion day and AWS user guide.

About the authors

Dr. Changsha Ma is an AI/ML Specialist at AWS. She is a technologist with a PhD in Computer Science, a master’s degree in Education Psychology, and years of experience in data science and independent consulting in AI/ML. She is passionate about researching methodological approaches for machine and human intelligence. Outside of work, she loves hiking, cooking, hunting food, and spending time with friends and families.

Ajjay Govindaram is a Senior Solutions Architect at AWS. He works with strategic customers who are using AI/ML to solve complex business problems. His experience lies in providing technical direction as well as design assistance for modest to large-scale AI/ML application deployments. His knowledge ranges from application architecture to big data, analytics, and machine learning. He enjoys listening to music while resting, experiencing the outdoors, and spending time with his loved ones.

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the ML data preparation for SageMaker Canvas and SageMaker Data Wrangler, with 15 years of experience building customer-centric and data-driven products.

Read More