Posts in category: Amazon AWS

Amazon Lex provides the power of automatic speech recognition (ASR) for converting speech to text, along with natural language understanding (NLU) for recognizing user intents. This combination allows you to develop sophisticated conversational interfaces using both voice and text for chatbots, IVR bots, and voicebots.

This week, we’re announcing Amazon Lex support for British English and US Spanish. With British English, your conversational bots can be localized to understand the British English accent, while delivering responses in Amazon Polly voices designed to sound like UK English speakers. Brilliant!

With support for US Spanish, you can develop applications for the second-most widely spoken language in the United States. Amazon Lex can now accurately recognize written and spoken Spanish, while providing responses using Amazon Polly’s natural sounding US Spanish voices. Listo!

In this post, we consider a hypothetical appliance manufacturer in the United States. Customers calling into their service center to schedule or change a repair appointment may prefer to speak in English or Spanish. The application for this use case allows callers to select their preferred language by saying “English” or “Español” when prompted. You want to provide them with the best customer service experience, no matter whether it is their washing machine or their lavadora that needs repair.

We show you how to create an Amazon Connect call center experience that supports both US English and US Spanish. Customers can schedule, change, and cancel appointments, using a fully automated solution that converses with them in their preferred language.

Building a multi-language conversational experience

This post uses the following sample conversations:

Agent: Thank you for calling. To continue in English, say “English,” for Spanish, say “Español.”

US English

Agent: I can schedule or change a repair appointment. How can I help?

User: I want to get my dishwasher fixed.

Agent: What city are you in?

User: Philadelphia

Agent: I have technicians available next week. When would you prefer to have them visit?

User: September 24th at noon

Agent: OK, you are all set for your dishwasher repair in Philadelphia on the 24th of September at noon.

US Spanish

Agent: Puede reservar o cambiar una cita de reparación. ¿Cómo puedo ayudar?

User: Me gustaria programar una cita

Agent: Para qué tipo de aparato?

User: Refrigerador

Agent: En que ciudad estas?

User: Brooklyn

Agent: En que fecha te gustaria que vinieran?

User: 24 de Septiembre

Agent: Bien, ya está todo listo para la reparación de su refrigerador en 24 de Septiembre, 2020.

To support these conversation models, you need to create Lex bots with relevant user intents. In this post, we create intents for ScheduleAppointment, ModifyAppointment, and CancelAppointment (and in Spanish, ReservarCita, ModificarCita, CancelarCita).

Deploying the sample Lex bots

To create the sample bots, perform the following steps. For this post, you create two Amazon Lex bots: AppointmentBot_enUS for US English, and AppointmentBot_esUS for US Spanish. To follow along with this post, you can create these Lex bots yourself on the Amazon Lex console, or import them directly.

1. To import the bots, download the US English bot and the US Spanish bot
2. On the Amazon Lex console, choose
3. Select the zip that you downloaded, and choose Import.
4. When the import process is complete, choose AppointmentBot_enUS, and choose
5. When the build it complete, go back to the Amazon Lex console main window, and choose Import.
6. Select the zip that you downloaded, and choose Import.
7. When the import process is complete, choose AppointmentBot_esUS, and choose

At this point, you should have two working Lex bots: one for US English, and one for US Spanish.

Creating your Amazon Connect instance

In this section, we integrate the bots with an Amazon Connect cloud-based call center instance. The first step is to create the Amazon Connect instance:

1. On the AWS Management Console, choose Amazon Connect.
2. If this is your first Amazon Connect instance, choose Get started; otherwise, choose Add an instance.
3. For Identity management, choose Store users within Amazon Connect.
4. Enter a URL prefix, such as appointment-bot-############, where ############ is your current AWS account number.
5. Choose Next step.
6. For Create an administrator, enter a name, password, and email address.
7. Choose Next step.
8. For Telephony Options, leave both call options selected by default.
9. Choose Next step.
10. For Data storage, choose Next step.
11. Review the settings and choose Create instance.

Associating your bots with your Amazon Connect instance

Now that you have an Amazon Connect instance, you can claim a phone number, create a contact flow, and integrate your contact flow with the two Lex bots you created in the prior step. First, associate your bots with your Amazon Connect instance:

1. On the Amazon Connect console, open your instance by choosing the Instance Alias
2. Choose Contact flows.
3. From the drop-down list, choose AppointmentBot_enUS. If you don’t see the bot in the list, make sure you have selected the same Region you used when you created your Lex bot.
4. Choose + Add Lex Bot.
5. From the drop-down list, choose AppointmentBot_esUS and choose + Add Lex Bot.

Configuring Amazon Connect to work with your bot

Now you can use your bots with Amazon Connect. First, claim a phone number for your Amazon Connect instance:

1. On the Amazon Connect console, choose Overview.
2. Choose the Amazon Connect
3. Choose the Login URL link, and enter the user name and password you specified earlier.
4. On the Amazon Connect console, for Step 1, Choose Begin.
5. For your phone number, choose a country, Direct Dial or Toll Free, and a phone number.
6. Choose Next.
7. If you want to test your new phone number, try it on the next screen or choose Skip for now.

For this post, you can skip the hours of operation, creating queues, and creating prompts. For more information on these features, see the Amazon Connect Administrator Guide. Now let’s import the contact flow.

8. Download and unzip the sample Amazon Connect contact flow for this post: manage_repairs.zip.
9. On the Amazon Connect console, go to Step 5, Create contact flows, choose View contact flows.
10. Choose Create contact flow.
11. From the drop-down at the top right side of the page, choose Import flow (beta).
12. Choose Select and select the manage-repairs.json file you downloaded, and select Import.
13. Choose Save, and then Publish.

Your contact flow should look like the following screenshot.

14. Choose the Routing icon from the side menu, and choose Phone numbers.
15. Choose your phone number to edit it, and change the contact flow or IVR to the Manage Repairs contact flow you just created.
16. Choose Save.

Your Amazon Connect instance is now configured to work with your Amazon Lex bots. Try calling the phone number to see how it works!

Conclusion

With the addition of British English and US Spanish language support, along with US English and Australian English, Amazon Lex allows you to create bots that can converse with users in their native language. You can combine Amazon Lex with Amazon Connect to create streamlined, multi-language call center user experiences in minutes. The additional language support in Amazon Lex is available at the same price, and in the same Regions, as US English. You can try these languages via the console, the AWS Command Line Interface (AWS CLI), and the AWS SDKs.

About the Authors

Claire Mitchell is a Design Consultant with the AWS Professional Services Conversational AI team. Occasionally she spends time exploring speculative design practices, textiles, and playing the drums.

Brian Yost is a Senior Consultant with the AWS Professional Services Conversational AI team. In his spare time, he enjoys mountain biking, home brewing, and tinkering with technology.

As a Product Manager on the Amazon Lex team, Harshal Pimpalkhute spends his time trying to get machines to engage (nicely) with humans.

Read More

University of Illinois, Urbana Champaign (UIUC) has partnered with the Amazon Machine Learning Solutions Lab to help UIUC football coaches prepare for games more efficiently and improve their odds of winning.

Previously, coaches prepared for games by creating a game planning sheet that only featured types of plays for a certain down and distance, and where the team was on the field. As a result, the coaching staff might miss important scenarios and opportunities. Additionally, preparing a game planning sheet was a manual process, with new data for each game being entered into a template each week, which is time-consuming and not scalable.

To add more insights to the current call sheet templates and help coaches prepare for games better, the team combined UIUC’s deep expertise in college football and coaching with the machine learning (ML) capabilities of Amazon SageMaker to create a state-of-the-art ML model that predicts the result of UIUC’s football plays. In addition, UIUC coaches now have an auto-generated visual game planning sheet based on key features that the model recommends. This gives them more insights on their strategy for the game and reduces the time it takes to generate the visual game planning sheets from 2.5 hours to less than 30 seconds.

“The UIUC Athletic department collaborated with the Amazon ML Solutions Lab to harness the power of machine learning to derive data-driven insights on what features to include in our planning and preparation for our football games,” says Kingsley Osei-Asibey, Director of Analytics & Football Technology at UIUC. “By selecting AWS as our primary ML/AI platform, we got to work alongside the experts at the ML Solutions Lab to create new and interesting insights using Amazon SageMaker. Now, all the manual analysis of data from past games that took us hours is automated, and our Fighting Illini coaches can generate weekly visual game planning sheets against different opponents at the press of a button.”

This post looks at how the Amazon ML Solutions Lab used features related to the plays during a football game to predict the result of a play, and then used the XGBoost importance score feature and correlation analysis to recommend features for coaches to analyze.

We provide code snippets to show you how we used the Amazon SageMaker XGBoost library to generate feature importance scores.

Data and model

We used UIUC’s game data from the 2018–2019 college football season, covering 24 features including in-game statistics, location of the play, UIUC’s strategies, and their opponent’s play types. We used those features to train an XGBoost model to predict if an offensive play will result in a win or loss. The UIUC coaches decided whether it’s a win or loss for a play based on different situations.

We then used the feature importance scores to select key features. We used the model for feature-selection purposes to recommend important scenarios represented by features. We selected XGBoost because it performs well on features with complex distributions, and it outputs feature importance scores to help us with feature selection and model interpretation.

The main goal was to generate game planning sheets for football coaches to use in games to give them an edge. We used the features from a well performant ML model trained to classify successful and unsuccessful plays to inform coaches and generate game planning sheets.

The following diagram summarizes the modeling steps taken to generate the ML-based features for the game planning sheet.

The rows are shuffled and split into five non-overlapping folds, which are then further split into training and validation sets. The training sets of each fold are balanced using the Synthetic Minority Oversampling Technique (SMOTE) algorithm.

Each fold includes the following steps:

1. Calculate a new feature score:
   1. Train an XGBoost model on the balanced training data set and extract the feature importances feat_i.
   2. Compute the Pearson’s correlation of the features and label in the balanced training dataset corr_i.
   3. Compute a new feature score as the product of absolute correlation and feature importance feature_score_i = feat_i * abs(corr_i).
2. Sort the features based on the feat_score.
3. Train multiple XGBoost models using the top 5 features, top 10 features, and so on, and evaluate validation balanced accuracy for each model.
4. Choose the best-performing model.

After we trained models from each of the five folds, we merged the important features. A feature is selected for the game planning sheet if it appears in the top 10 features (ranked by feature importance score) of at least three folds.

Calculating the new feature score

In the previous section, we described the construction of a new feature score. This new feature score incorporates the feature importance from a non-linear XGBoost model, as well as direct linear correlation. The purpose of this new feature score is to select features that are relevant to winning or losing a play. A feature with a high feature score has high XGBoost feature importance and high linear correlation with the label, making it a relevant feature for game planning sheets.

In this section, we dive deeper into the construction of the new feature score with code snippets. The feature score is a combination of feature importance from a trained XGBoost model and linear correlation of the features and the label.

First, we train an XGBoost model using Amazon SageMaker built-in algorithms. Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models quickly. Amazon SageMaker provides several built-in algorithms (such as XGBoost) for a variety of problem types.

This trained XGBoost model provides a first look into which features are important to the UIUC football team winning a play. See the following code:

from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(region, "xgboost", "0.90-1")

hyperparameters = {
    "max_depth":"7",
    "eta":"0.01",
    "gamma":"3",
    "min_child_weight":"6",
    "subsample":"0.6",
    "silent":"0",
    "objective":"binary:logistic",
    "num_round":"330"
}

instance_type = 'ml.m5.2xlarge'
output_path = "s3://{}/{}/{}/output".format(bucket, "model", "xgboost")

job_name = "xgboost-".format(i+1) + time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())

estimator = sagemaker.estimator.Estimator(
    container, 
    role, 
    hyperparameters=hyperparameters,
    train_instance_count=1, 
    train_instance_type=instance_type, 
    train_volume_size=5,
    output_path=output_path, 
    sagemaker_session=sagemaker.Session()
)

train_input = sagemaker.s3_input(
    s3_data="s3://{}/{}/{}".format(bucket, "train", "balanced_train_data.csv"), 
    content_type='csv'
)
estimator.fit({"train": train_input}, job_name=job_name)

Amazon SageMaker stores the model object in the specified Amazon Simple Storage Service (Amazon S3) bucket. To calculate the feature score, we need to download model.tar.gz from Amazon S3 to our Amazon SageMaker notebook instance. See the following code:

model_path = "s3://{}/{}/{}/output/{}".format(
    bucket, "model", "xgboost",
    "xgboost-2019-06-16-09-56-39-854/output/model.tar.gz"
)

fs = s3fs.S3FileSystem()

with fs.open(model_path, "rb") as f:
    with tarfile.open(fileobj=f, mode="r") as tar_f:
        with tar_f.extractfile("xgboost-model") as extracted_f:
            xgbooster = pickle.load(extracted_f)

Finally, we calculate the new feature score as feature_score_i = feat_i * abs(corr_i). We use the absolute value of the correlation because our goal is to find features that are relevant to winning or losing a play, and a highly negative correlation indicates a strong linear relationship between the feature and the UIUC football team losing the play. See the following code:

#the xgbooster object replaces the original feature names with 'f0,...f'
#here we create a mapping to obtain the original feature names
feature_name_map = dict(zip([f"f{i}" for i in range(len(feature_names))], feature_names))

features_importance_df = pd.DataFrame([xgbooster.get_fscore()], index=["weight"]).T
features_importance_df["normalized_weight"] = features_importance_df["weight"]/features_importance_df["weight"].sum()
feature_importances_df["feature_name"] = feature_importances_df.index.map(feature_name_map)

correlation_df = pd.DataFrame(balanced_train_data_df[FEATURES].corr()[LABEL])
correlation_df["absolute_corr"] = correlation_df[LABEL].abs()

feature_score_df = pd.merge(
    features_importance_df, correlation_df.reset_index(), 
    left_on="feature_name", right_on="index"
)

feature_score_df["feature_score"] = feature_score_df["absolute_corr"] * feature_score_df["normalized_weight"]

The following graph shows a plot of feature_score vs.rank for each fold. High values on the y-axis indicate that the feature was important for the XGBoost model and has high correlation with winning or losing a play. The key takeaway from this plot is additional features after feature number 105 don’t add any new information, and the optimum number of features to use lies between 0–105.

Evaluating the model

We performed five-fold cross-validation on the XGBoost model, and compared it to three baseline models: a model predicting every sample as lost, a model predicting every sample as win, and a random model assigning win or loss with a 50/50 chance.

Because the dataset is imbalanced with 56% of the plays labeled as lost and 44% as won, we used the weighted accuracy metrics considering the class weights when comparing our model to the naïve baselines. The weighted accuracy for all three naïve baselines is 50%, and average weighted accuracy of the XGBoost is 65.2% across five folds, which shows that our model has 15% improvement compared to the baselines.

The following plot shows validation balanced accuracy vs. the number of top features for each fold. For each data point, an XGBoost model is trained using the top n features, where n is the value on the x-axis, and evaluated on the fold’s validation dataset to obtain the validation balanced accuracy. The top performing model for each is annotated in the plot. For example, Fold 0’s best-performing model uses the top 60 features (as determined in the preceding plot), which has a validation balanced accuracy of 64.8%. Features ranked above 105 aren’t evaluated because the previous plot shows that features ranked above 105 contribute little information.

The following table summarizes the results of the procedure we outlined. For each fold, the balanced accuracy performance improves after performing feature selection, with an average increase of 3.2%.

To further improve the models, we used Amazon SageMaker automated model tuning for hyperparameter optimization. We used the best features identified in the preceding step for each fold, and performed 20 iterations of Bayesian optimization on each fold.

Feature selection and game planning sheet recommendation across five folds

The end goal is to create a new game planning sheet using features derived from the XGBoost models. A high-performing model indicates that the extracted features are relevant to winning a play. The output of the training stage results in an XGBoost model for each fold. A feature is selected for the game planning sheet if it appears in the top 10 features (ranked by feature importance score) of at least three folds.

After reviewing these features with the UIUC coaching staff, the coaches designed new game planning sheets to analyze the best play types based on how their opponent would be playing defense. These additional features will help the coaches prepare more scenarios before the games start, and players can react faster and more accurately against opponents.

Summary

UIUC football coaches partnered with the Amazon ML Solutions Lab and created an ML model to gain more insights on their performance and strategies. This solution also saves the coaches’ time when preparing for a game; instead of manually analyzing the best plays to call under different situations, coaches can automate this process using the features the ML model recommends.

This model is customized for UIUC’s football team and their opponents, and will help UIUC’s coaches prepare for more scenarios in upcoming seasons. Additionally, it will help players react correctly and quickly to game situations.

If you’d like help accelerating the use of ML in your products and services, please contact the Amazon ML Solutions Lab program.

About the Authors

 Ninad Kulkarni is a Data Scientist in the Amazon Machine Learning Solutions Lab. He helps customers adopt ML and AI by building solutions to address their business problems. Most recently, he has built predictive models for sports and automotive customers.

Daliana Zhen Liu is a Data Scientist in the Amazon Machine Learning Solutions Lab. She has built ML models to help customers accelerate their business in sports, media and education. She is passionate about introducing data science to more people.

Tianyu Zhang is a Data Scientist in the Amazon Machine Learning Solutions Lab. He helps customers solve business problems by applying ML and AI techniques. Most recently, he has built NLP model and predictive model for procurement and sports.

Read More

Amazon Comprehend is a natural language processing (NLP) service that uses machine learning (ML) to find insights and relationships like people, places, sentiments, and topics in unstructured text. You can now use Amazon Comprehend ML capabilities to detect and redact personally identifiable information (PII) in customer emails, support tickets, product reviews, social media, and more. No ML experience required. For example, you can analyze support tickets and knowledge articles to detect PII entities and redact the text before you index the documents in the search solution. After that, search solutions are free of PII entities in documents. Redacting PII entities helps you protect privacy and comply with local laws and regulations.

Customer use case: TeraDact Solutions

TeraDact Solutions has already put this new feature to work. TeraDact Solutions’ software offers a robust alternative for secure information sharing in a world of ever-increasing compliance and privacy concerns. With its signature Information Identification & Presentation (IIaP) capabilities, TeraDact’s tools provide the user with a safe information sharing environment. “Using Amazon Comprehend for PII redaction with our tokenization system not only helps us reach a larger set of our customers but also helps us overcome the shortcomings of rules-based PII detection which can result in false alarms or missed details. PII detection is critical for businesses and with the power of context-aware NLP models from Comprehend we can uphold the trust customers place in us with their information. Amazon is innovating in ways to help push our business forward by adding new features which are critical to our business thereby providing enhanced service to 100% of customers able to access Comprehend in AWS.” said Chris Schrichte, CEO, TeraDact Solutions, Inc.

In this post, I cover how to use Amazon Comprehend to detect PII and redact the PII entities via the AWS Management Console and the AWS Command Line Interface (AWS CLI).

Detecting PII in Amazon Comprehend

When you analyze text using Amazon Comprehend real-time analysis, Amazon Comprehend automatically identifies PII, as summarized in the following table.

For each detected PII entity, you get the type of PII, a confidence score, and begin and end offset. These offsets help you locate PII entities in your documents for document processing to redact it at the secure storage or downstream solutions.

Analyzing text on the Amazon Comprehend console

To get started with Amazon Comprehend, all you need is an AWS account. To use the console, complete the following steps:

1. On the Amazon Comprehend console, in the Input text section, select Built-in.
2. For Input text, enter your text.
3. Choose Analyze.

4. On the Insights page, choose the PII

The PII tab shows color-coded text to indicate different PII entity types, such as name, email, address, phone, and others. The Results section shows more information about the text. Each entry shows the PII entity, its type, and the level of confidence Amazon Comprehend has in this analysis.

Analyzing text via the AWS CLI

To perform real-time analysis using the AWS CLI, enter the following code:

aws comprehend detect-pii-entities 
--language-code en 
--text 
" Good morning, everybody. My name is Van Bokhorst Serdar, and today I feel like sharing a whole lot of personal information with you. Let's start with my Email address SerdarvanBokhorst@dayrep.com. My address is 2657 Koontz Lane, Los Angeles, CA. My phone number is 818-828-6231. My Social security number is 548-95-6370. My Bank account number is 940517528812 and routing number 195991012. My credit card number is 5534816011668430, Expiration Date 6/1/2022, my C V V code is 121, and my pin 123456. Well, I think that's it. You know a whole lot about me. And I hope that Amazon comprehend is doing a good job at identifying PII entities so you can redact my personal information away from this document. Let's check."

To view the output, open the JSON response object and look at the detected PII entities. For each entity, the service returns the type of PII, confidence score metric, BeginOffset, and EndOffset. See the following code:

{
    "Entities": [
        {
            "Score": 0.9996334314346313,
            "Type": "NAME",
            "BeginOffset": 36,
            "EndOffset": 55
        },
        {
            "Score": 0.9999902248382568,
            "Type": "EMAIL",
            "BeginOffset": 167,
            "EndOffset": 195
        },
        {
            "Score": 0.9999983310699463,
            "Type": "ADDRESS",
            "BeginOffset": 211,
            "EndOffset": 245
        },
        {
            "Score": 0.9999997615814209,
            "Type": "PHONE",
            "BeginOffset": 265,
            "EndOffset": 277
        },
        {
            "Score": 0.9999996423721313,
            "Type": "SSN",
            "BeginOffset": 308,
            "EndOffset": 319
        },
        {
            "Score": 0.9999984502792358,
            "Type": "BANK_ACCOUNT_NUMBER",
            "BeginOffset": 347,
            "EndOffset": 359
        },
        {
            "Score": 0.9999974966049194,
            "Type": "BANK_ROUTING",
            "BeginOffset": 379,
            "EndOffset": 388
        },
        {
            "Score": 0.9999991655349731,
            "Type": "CREDIT_DEBIT_NUMBER",
            "BeginOffset": 415,
            "EndOffset": 431
        },
        {
            "Score": 0.9923601746559143,
            "Type": "CREDIT_DEBIT_EXPIRY",
            "BeginOffset": 449,
            "EndOffset": 457
        },
        {
            "Score": 0.9999997615814209,
            "Type": "CREDIT_DEBIT_CVV",
            "BeginOffset": 476,
            "EndOffset": 479
        },
        {
            "Score": 0.9998345375061035,
            "Type": "PIN",
            "BeginOffset": 492,
            "EndOffset": 498
        }
    ]
}

Asynchronous PII redaction batch processing on the Amazon Comprehend console

You can redact documents by using Amazon Comprehend asynchronous operations. You can choose redaction mode Replace with PII entity to replace PII entities with PII entity type, or choose to mask PII entity with redaction mode Replace with character and replace the characters in PII entities with a character of your choice (!, #, $, %, &, *, or @).

To analyze and redact large documents and large collections of documents, ensure that the documents are stored in an Amazon Simple Storage Service (Amazon S3) bucket and start an asynchronous operation to detect and redact PII in the documents. The results of the analysis are returned in an S3 bucket.

1. On the Amazon Comprehend console, choose Analysis jobs.
2. Choose Create job.

3. On the Create analysis job page, for Name, enter a name (for this post, we enter comprehend-blog-redact-01).
4. For Analysis type, choose Personally identifiable information (PII).
5. For Language, choose English.

6. In the PII detection settings section, for Output mode, select Redactions.
7. Expand PII entity types and select the entity types to redact.
8. For Redaction mode, choose Replace with PII entity type.

Alternatively, you can choose Replace with character to replace PII entities with a character of your choice (!, #, $, %, &, *, or @).

9. In the Input data section, for Data source, select My documents.
10. For S3 location, enter the S3 path for pii-s3-input.txt.

This text file has the same example content we used earlier for real-time analysis.

11. In the Output data section, for S3 location, enter the path to the output folder in Amazon S3.

Make sure you choose the correct input and output paths based on how you organized the document.

12. In the Access permissions section, for IAM role, select Create an IAM role.

You need an AWS Identity and Access Management (IAM) role with required permissions to access the input and output S3 buckets for the job that is created and propagated.

13. For Permissions to access, choose Input and Output S3 buckets.
14. For Name suffix, enter a suffix for your role (for this post, we enter ComprehendPIIRole).
15. Choose Create job.

You can see the job comprehend-blog-redact-01 with the job status In progress.

When the job status changes to Completed, you can access the output file to view the output. The pii-s3-input.txt file has the same example content we used earlier, and using redaction mode replaces PII with its PII entity type. Your output looks like the following text:

If you have very long entity types, you may prefer to mask PII with a character. If you choose to replace PII with the character *, your output looks like the following text. :

Asynchronous PII redaction batch processing via the AWS CLI

To perform the PII redaction job using the AWS CLI, enter the following code:

aws comprehend start-pii-entities-detection-job 
 --input-data-config S3Uri="s3://ai-ml-services-lab/public/labs/comprehend/pii/input/redact/pii-s3-input.txt"  
 --output-data-config S3Uri="s3://ai-ml-services-lab/public/labs/comprehend/pii/output/redact/"  
 --mode "ONLY_REDACTION" 
 --redaction-config PiiEntityTypes="BANK_ACCOUNT_NUMBER","BANK_ROUTING","CREDIT_DEBIT_NUMBER","CREDIT_DEBIT_CVV","CREDIT_DEBIT_EXPIRY","PIN","EMAIL","ADDRESS","NAME","PHONE","SSN",MaskMode="REPLACE_WITH_PII_ENTITY_TYPE" 
 --data-access-role-arn "arn:aws:iam::<ACCOUNTID>:role/service-role/AmazonComprehendServiceRole-ComprehendPIIRole" 
 --job-name "comprehend-blog-redact-001" 
 --language-code "en"

The request yields the following output:

{
    "JobId": "e41101e2f0919a320bc0583a50f86b5f",
    "JobStatus": "SUBMITTED"
}

To monitor the job request, enter the following code:

aws comprehend describe-pii-entities-detection-job --job-id " e41101e2f0919a320bc0583a50f86b5f "

The following output shows that the job is complete:

{
    "PiiEntitiesDetectionJobProperties": {
        "JobId": "e41101e2f0919a320bc0583a50f86b5f",
        "JobName": "comprehend-blog-redact-001",
        "JobStatus": "COMPLETED",
        "SubmitTime": <SubmitTime>,
        "EndTime": <EndTime>,
        "InputDataConfig": {
            "S3Uri": "s3://ai-ml-services-lab/public/labs/comprehend/pii/input/redact/pii-s3-input.txt",
            "InputFormat": "ONE_DOC_PER_LINE"
        },
        "OutputDataConfig": {
            "S3Uri": "s3://ai-ml-services-lab/public/labs/comprehend/pii/output/redact/<AccountID>-PII-e41101e2f0919a320bc0583a50f86b5f/output/"
        },
        "RedactionConfig": {
            "PiiEntityTypes": [
                "BANK_ACCOUNT_NUMBER",
                "BANK_ROUTING",
                "CREDIT_DEBIT_NUMBER",
                "CREDIT_DEBIT_CVV",
                "CREDIT_DEBIT_EXPIRY",
                "PIN",
                "EMAIL",
                "ADDRESS",
                "NAME",
                "PHONE",
                "SSN"
            ],
            "MaskMode": "REPLACE_WITH_PII_ENTITY_TYPE"
        },
        "LanguageCode": "en",
        "DataAccessRoleArn": "arn:aws:iam::<AccountID>:role/ComprehendBucketAccessRole",
        "Mode": "ONLY_REDACTION"
    }
}

After the job is complete, the output file is plain text (same as the input file). Other Amazon Comprehend asynchronous jobs (start-entities-detection-job) have an output file called output.tar.gz, which is a compressed archive that contains the output of the operation. Start-pii-entities-detection-job retains the folder and file structure as input. Our comprehend-blog-redact-001 job input file pii-s3-input.txt has the respective pii-s3-input.txt.out file with the redacted text in the jobs output folder. You can find the Amazon S3 location in the output from monitoring the job; the JSON element PiiEntitiesDetectionJobProperties.OutputDataConfig.S3uri has the file pii-s3-input.txt.out and the redacted content with PII entity type.

Conclusion

As of this writing, the PII detection feature in Amazon Comprehend is available for US English in the following Regions:

• US East (Ohio)
• US East (N. Virginia)
• US West (Oregon),
• Asia Pacific (Mumbai)
• Asia Pacific (Seoul)
• Asia Pacific (Singapore)
• Asia Pacific (Sydney)
• Asia Pacific (Tokyo)
• EU (Frankfurt)
• EU (Ireland)
• EU (London)
• AWS GovCloud (US-West)

Take a look at the pricing page, give the feature a try, and please send us feedback either via the AWS forum for Amazon Comprehend or through your usual AWS support contacts.

About the Author

Sriharsha M S is an AI/ML specialist solution architect in the Strategic Specialist team at Amazon Web Services. He works with strategic AWS customers who are taking advantage of AI/ML to solve complex business problems. He provides technical guidance and design advice to implement AI/ML applications at scale. His expertise spans application architecture, bigdata, analytics and machine learning.

Read More

The volume of user-generated content (UGC) and third-party content has been increasing substantially in sectors like social media, ecommerce, online advertising, and photo sharing. However, such content needs to be reviewed to ensure that end-users aren’t exposed to inappropriate or offensive material, such as nudity, violence, adult products, or disturbing images. Today, some companies simply react to user complaints to take down offensive images, ads, or videos, whereas many employ teams of human moderators to review small samples of content. However, human moderators alone can’t scale to meet these needs, leading to a poor user experience or even a loss of brand reputation.

With Amazon Rekognition, you can automate or streamline your image and video analysis workflows using machine learning (ML). Amazon Rekognition provides an image moderation API that can detect unsafe or inappropriate content containing nudity, suggestiveness, violence, and more. You get a hierarchical taxonomy of labels that you can use to define your business rules, without needing any ML experience. Each detection by Amazon Rekognition comes with a confidence score between 0–100, which provides a measure of how confident the ML model is in its prediction.

Content moderation still requires human reviewers to audit results and judge nuanced situations where AI may not be certain in its prediction. Combining machine predictions with human judgment and managing the infrastructure needed to set up such workflows is hard, expensive, and time-consuming to do at scale. This is why we built Amazon Augmented AI (Amazon A2I), which lets you implement a human review of ML predictions and is directly integrated with Amazon Rekognition. Amazon A2I allows you to use in-house, private, or third-party vendor workforces with a web interface that has instructions and tools they need to complete their review tasks.

You can easily set up the criteria that triggers a human review of a machine prediction; for example, you can send an image for further human review if Amazon Rekognition’s confidence score is between 50–90. Amazon Rekognition handles the bulk of the work and makes sure that every image gets scanned, and Amazon A2I helps send the remaining content for further review to best utilize human judgment. Together, this helps ensure that you get full moderation coverage while maintaining very high accuracy, at a fraction of the cost to review each image manually.

In this post, we show you how to use Amazon Rekognition image moderation APIs to automatically detect explicit adult, suggestive, violent, and disturbing content in an image and use Amazon A2I to onboard human workforces, set up human review thresholds of the images, and define human review tasks. When these conditions are met, images are sent to human reviewers for further review, which is performed according to the instructions in the human review task definition.

Prerequisites

This post requires you to complete the following prerequisites:

• Create an AWS Identity and Access Management (IAM) role. To create a human review workflow, you need to provide an IAM role that grants Amazon A2I permission to access Amazon Simple Storage Service (Amazon S3) for reading objects to render in a human task UI and writing the results of the human review. This role also needs an attached trust policy to give Amazon SageMaker permission to assume the role. This allows Amazon A2I to perform actions in accordance with permissions that you attach to the role. For example policies that you can modify and attach to the role you use to create a flow definition, see Add Permissions to the IAM Role Used to Create a Flow Definition.
• Configure permission to invoke the Amazon Rekognition DetectModerationLabels You need to attach the AmazonRekognitionFullAccess policy to the AWS Lambda function that calls the Amazon Rekognition detect_moderation_labels API.
• Provide Amazon S3 Access, Put, and Get permission to Lambda if you wish to have Lambda use Amazon S3 to access images for analysis.
• Give the Lambda function AmazonSageMakerFullAccess access to the Amazon A2I services for the human review.

Creating a private work team

A work team is a group of people that you select to review your documents. You can create a work team from a workforce, which is made up of Amazon Mechanical Turk workers, vendor-managed workers, or your own private workers that you invite to work on your tasks. Whichever workforce type you choose, Amazon A2I takes care of sending tasks to workers. For this post, you create a work team using a private workforce and add yourself to the team to preview the Amazon A2I workflow.

To create your private work team, complete the following steps:

1. Navigate to the Labeling workforces page on the Amazon SageMaker console.
2. On the Private tab, choose Create private team.
3. For Team name, enter an appropriate team name.
4. For Add workers, you can choose to add workers to your workforce by importing workers from an existing user group in AWS Cognito or by inviting new workers by email.

For this post, we suggest adding workers by email. If you create a workforce using an existing AWS Cognito user group, be sure that you can access an email in that workforce to complete this use case.

5. Choose Create private team.
   
6. On the Private tab, choose the work team you just created to view your work team ARN.
7. Record the ARN to use when you create a flow definition in the next section.

After you create the private team, you get an email invitation. The following screenshot shows an example email.

8. Choose the link to log in and change your password.

You’re now registered as a verified worker for this team. The following screenshot shows the updated information on the Private tab.

Your one-person team is now ready, and you can create a human review workflow.

Creating a human review workflow

In this step, you create a human review workflow, where you specify your work team, identify where you want output data to be stored in Amazon S3, and create instructions to help workers complete your document review task.

To create a human review workflow, complete the following:

1. In the Augmented AI section on the Amazon SageMaker console, navigate to the Human review workflows
2. Choose Create human review workflow.

On this page, you configure your workflow.

3. Enter a name for your workflow.
4. Choose an S3 bucket where you want Amazon A2I to store the output of the human review.
5. Choose an IAM role for the workflow.

You can create a new role automatically with Amazon S3 access and an Amazon SageMaker execution policy attached, or you can choose a role that already has these permissions attached.

6. In the Task type section, select Rekognition – Image moderation.
   
7. In the Amazon Rekognition-Image Moderation – Conditions for invoking human review section, you can specify conditions that trigger a human review.

For example, if the confidence of the output label produced by Amazon Rekognition is between the range provided (70–100, for this use case), the document is sent to the portal for human review. You can also select different confidence thresholds for each image moderation output label through Amazon A2I APIs.

8. In the Worker task template creation section, if you already have an A2I worker task template, you can choose Use your own template. Otherwise, select Create from a default template and enter a name and task description. For this use case, you can use the default worker instructions provided.
9. In the Workers section, select Private.
10. For Private teams, choose the private work team you created earlier.
    
11. Choose Create.

You’re redirected to the Human review workflows page, where you can see the name and ARN of the human review workflow you just created.

12. Record the ARN to use in the next section.
    

Configuring Lambda to run Amazon Rekognition

In this step, you create a Lambda function to call the Amazon Rekognition API detect_moderation_labels. You use the HumanLoopConfig parameter of detect_moderation_labels to integrate an Amazon A2I human review workflow into your Amazon Rekognition image moderation job.

1. On the Lambda console, create a new function called A2IRegok.
2. For Runtime, choose Python 3.7.
3. Under Permission, choose Use an existing role.
4. Choose the role you created.
5. In the Function code section, remove the function code and replace it with the following code.
   1. Inside the Lambda function, import two libraries: uuid and boto3.
   2. Modify the function code as follows:
      1. Replace the FlowDefinationArn in line 12 with one you saved in the last step.
      2. On line 13, provide a unique name to the HumanLoopName or use uuid to generate a unique ID.
      3. You use the detect_moderation_labels API operation to analyze the picture (JPG, PNG). To use the picture from the Amazon S3 bucket, specify the bucket name and key of the file inside the API call as shown in lines 7 and 8.

  1 import boto3
  2 import uuid
  3
  4 def lambda_handler(event, context):
  5     if event:
  6
  7      bucket_name = "a2idemorekog". # Add your sourcebucketname
  8      src_filename = "1.png".       # Add the src filename
  9      rekognition = boto3.client('rekognition')
 10      human_loop_unique_id = str(uuid.uuid4()) + '1'
 11      humanLoopConfig = {
 12                 'FlowDefinitionArn':"arn:aws:sagemaker:us-east-1:123456789123:flow-definition/a2i-rekognition-wf",
 13                 'HumanLoopName':human_loop_unique_id
 14             }
 15
 16      response = rekognition.detect_moderation_labels(
 17                 Image = {
 18                     "S3Object": {
 19                      "Bucket": bucket_name,
 20                      "Name": src_filename,
 21                      }
 22                     },
 23      HumanLoopConfig = humanLoopConfig
 24             )

Calling Amazon Rekognition using Lambda

To configure and run a serverless function, complete the following steps:

1. On the Lambda console, choose your function.
2. Choose Configure test events from the drop-down menu.

The editor appears to enter an event to test your function.

3. On the Configure test event page, select Create new test event.
4. For Event template, choose hello-world.
5. For Event name, enter a name; for example, DemoEvent.
6. You can change the values in the sample JSON. For this use case, no change is needed.

For more information, see Run a Serverless “Hello, World!” and Create a Lambda function with the console.

7. Choose Create.
8. To run the function, choose Test.

When the test is complete, you can view the results on the console:

• Execution result – Verifies that the test succeeded
• Summary – Shows the key information reported in the log output
• Log output – Shows the logs the Lambda function generated

The response to this call contains the inference from Amazon Rekognition and the evaluated activation conditions that may or may not have led to a human loop creation. If a human loop is created, the output contains HumanLoopArn. You can track its status using the Amazon A2I API operation DescribeHumanLoop.

Completing a human review of your image

To complete a human review of your image, complete the following steps:

1. Open the URL in the email you received.

You see a list of reviews you are assigned to.

2. Choose the image you want to review.
3. Choose Start working.

After you start working, you must complete the task within 60 minutes.

4. Choose an appropriate category for the image.

Before choosing Submit, if you go to the Human review workflow page on the Amazon SageMaker console and choose the human review workflow you created, you can see a Human loops summary section for that workflow.

5. In your worker portal, when you’re done working, choose Submit.

After you complete your job, the status of the human loop workflow is updated.

If you navigate back to the Human review workflow page, you can see the human loop you just completed has the status Completed.

Processing the output

The output data from your review is located in Bucket when you configured your human review workflow on the Amazon A2I console. The path to the data uses the following pattern: YYYY/MM/DD/hh/mm/ss.

The output file (output.json) is structured as follows:

{
    "awsManagedHumanLoopRequestSource": "AWS/Rekognition/DetectModerationLabels/Image/V3",
    "flowDefinitionArn": "arn:aws:sagemaker:us-east-1:111122223333:flow-definition/a2i-rekog-blog",
    "humanAnswers": [
        {
            "answerContent": {
                "AWS/Rekognition/DetectModerationLabels/Image/V3": {
                    "moderationLabels": [
                        {
                            "name": "Weapon Violence",
                            "parentName": "Violence"
                        },
                        {
                            "name": "Violence",
                            "parentName": ""
                        }
                    ]
                }
            },
            "submissionTime": "2020-05-27T15:44:39.726Z",
            "workerId": "000cd1c234b5fcc7",
            "workerMetadata": {
                "identityData": {
                    "identityProviderType": "Cognito",
                    "issuer": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_00aa00a",
                    "sub": "b000a000-0b00-0ae0-bf00-0000f0bfd00d"
                }
            }
        }
    ],
    "humanLoopName": "389fd1a7-c658-4020-8f73-e9afcbfa8fd31",
    "inputContent": {
        "aiServiceRequest": {
            "humanLoopConfig": {
                "flowDefinitionArn": "arn:aws:sagemaker:us-east-1:111122223333:flow-definition/a2i-rekog-blog",
                "humanLoopName": "389fd1a7-c658-4020-8f73-e9afcbfa8fd31"
            },
            "image": {
                "s3Object": {
                    "bucket": "AWSDOC-EXAMPLE-BUCKET",
                    "name": "1.png"
                }
            }
        },
        "aiServiceResponse": {
            "moderationLabels": [
                {
                    "confidence": 80.41172,
                    "name": "Weapon Violence",
                    "parentName": "Violence"
                },
                {
                    "confidence": 80.41172,
                    "name": "Violence",
                    "parentName": ""
                }
            ],
            "moderationModelVersion": "3.0"
        },
        "humanTaskActivationConditionResults": {
            "Conditions": [
                {
                    "And": [
                        {
                            "ConditionParameters": {
                                "ConfidenceLessThan": 100,
                                "ModerationLabelName": "*"
                            },
                            "ConditionType": "ModerationLabelConfidenceCheck",
                            "EvaluationResult": true
                        },
                        {
                            "ConditionParameters": {
                                "ConfidenceGreaterThan": 60,
                                "ModerationLabelName": "*"
                            },
                            "ConditionType": "ModerationLabelConfidenceCheck",
                            "EvaluationResult": true
                        }
                    ],
                    "EvaluationResult": true
                }
            ]
        },
        "selectedAiServiceResponse": {
            "moderationLabels": [
                {
                    "confidence": 80.4117202758789,
                    "name": "Weapon Violence",
                    "parentName": "Violence"
                },
                {
                    "confidence": 80.4117202758789,
                    "name": "Violence",
                    "parentName": ""
                }
            ],
            "moderationModelVersion": "3.0"
        }
    }
}

In this JSON object, you have all the input and output content in one place so that you can parse one file to get the following:

• humanAnswers – Contains answerContent, which lists the labels chosen by the human reviewer, and workerMetadata, which contains information that you can use to track private workers
• inputContent – Contains information about the input data object that was reviewed, the label category options available to workers, and the responses workers submitted

For more information about the location and format of your output data, see Monitor and Manage Your Human Loop.

Conclusion

This post has merely scratched the surface of what Amazon A2I can do. Amazon A2I is available in 12 Regions. For more information, see Region Table. To learn more about the Amazon Rekognition DetectModerationLabels API integration with Amazon A2I, see Use Amazon Augmented AI with Amazon Rekognition.

For video presentations, sample Jupyter notebooks, or more information about use cases like document processing, object detection, sentiment analysis, text translation, and others, see Amazon Augmented AI Resources.

About the Author

Suresh Patnam is a Solutions Architect at AWS. He helps customers innovate on the AWS platform by building highly available, scalable, and secure architectures on Big Data and AI/ML. In his spare time, Suresh enjoys playing tennis and spending time with his family.

Read More