Monthly Archives: April 2023

Amazon Kendra is an intelligent search service powered by machine learning (ML). Amazon Kendra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they’re looking for, even when it’s scattered across multiple locations and content repositories within your organization.

Amazon Kendra FAQs allow users to upload frequently asked questions with their corresponding answers. This helps to consistently answer common queries among end-users. As of this writing, when you want to update FAQs, you must delete the FAQ and create it again. In this post, we present a simpler, faster approach for updating your Amazon Kendra FAQs (with versioning enabled). Our method eliminates the manual steps of creating and deleting FAQs when you update their contents.

Overview of solution

We use a fully deployable AWS CloudFormation template to create an Amazon Simple Storage Service (Amazon S3) bucket, which becomes the source to store your Amazon Kendra FAQs. Each index-based FAQ is maintained in the folder with a prefix relating to the Amazon Kendra index.

This solution uses an AWS Lambda function that gets triggered by an Amazon S3 event notification. When you upload an FAQ to the S3 folder mapped to a specific Amazon Kendra index, it creates a new version of the FAQ for your index. Older versions of FAQs are deleted only after the new FAQ index version is created, achieving near-zero downtime of index searching.

The following figure shows the workflow of how our method creates and deletes a new version of an Amazon Kendra FAQ.

The workflow steps are as follows:

1. The user uploads the Amazon Kendra FAQ document to the S3 bucket mapped to the Amazon Kendra index.
2. The Amazon S3 PutObject event triggers the Lambda function, which reads the event details.
3. The Lambda function creates a new version of the FAQ for the target index for each uploaded document and deletes the older versions of the FAQ.
4. The Lambda function then publishes a message to Amazon Simple Notification Service (Amazon SNS), which sends an email to the user notifying them that the FAQ has been successfully updated.

Prerequisites

Before you begin the walkthrough, you need an AWS account (if you don’t have one, you can sign up for one). You also need to create the files containing the sample FAQs:

• basic.csv – The following code is the sample FAQ CSV template:

  How many free clinics are in Spokane WA?, 13, https://www.freeclinics.com/
  How many free clinics are there in Mountain View Missouri?, 7, https://www.freeclinics.com/

• demo.json – The following code is the sample FAQ JSON template:

  {
    "SchemaVersion": 1,
    "FaqDocuments": [
      {
        "Question": "How many free clinics are in Spokane WA?",
        "Answer": "13"
      },
      {
        "Question": "How many free clinics are there in Mountain View Missouri?",
        "Answer": "7",
        "Attributes": {
          "_source_uri": "https://www.freeclinics.com",
          "_category": "Charitable Clinics"
        }
      }
    ]
  }

• header_demo.csv – The following code is the sample FAQ CSV template with header:

  _question,_answer,_last_updated_at
  How many free clinics are in Spokane WA?, 13, 2012-03-25T12:30:10+01:00
  How many free clinics are there in Mountain View Missouri?, 7, 2012-03-25T12:30:10+01:00

Deploy the solution

The CloudFormation templates that create the resources used by this solution can found in the GitHub repository. Follow the instructions in the repository to deploy the solution. AWS CloudFormation creates the following resources in your account:

• An S3 bucket that will be the source for the Amazon Kendra FAQ.
• An Amazon Kendra index.
• An AWS Identity and Access Management (IAM) role for the Amazon Kendra FAQ to read (GetObject) from the S3 bucket.
• A Lambda function that is configured to get triggered by an Amazon S3 event. The function is created outside of an Amazon VPC.

Note that resource creation can take approximately 30 minutes.

After you run the deployment, you’ll receive an email prompting you to confirm the subscription at the approver email address. Choose Confirm subscription.

You’re redirected to a page confirming your subscription.

Verify that the Amazon Kendra index is listed on the Amazon Kendra console. In this post, we named the Amazon Kendra index sample-kendra-index.

Upload a sample FAQ document to Amazon S3

In the previous step, you successfully deployed the CloudFormation stack. We use the output of the stack in the following steps:

1. On the Outputs tab of the CloudFormation stack, note the values for S3Bucket (kendra-faq-<random-stack-id>) and KendraIndex.
   
2. On the Amazon S3 console, navigate to the S3 bucket created from the CloudFormation stack.
3. Choose Create folder and create a folder called faq-<index-id>. For index-id, use the value you noted for the CloudFormation parameter KendraIndex. After the folder is created, this becomes the prefix for the sample-kendra-index FAQ.
   
4. Upload the demo.json FAQ document to that folder.
   

Verify that the index FAQ is created

To confirm that the index FAQ is created, complete the following steps:

1. On the Amazon Kendra console, navigate to the index sample_kendra_index, which was created as part of the deployment.
2. Navigate to the FAQs page for this index to check if an FAQ is listed.

The index has the naming convention <file-name>-faq-<Date-Time>.

When the FAQ is successfully created, you will receive another email informing you about it. You may upload new versions of the FAQ after you have received this email.

Note that the automation identifies the file format that it must use while creating the FAQ by reading the uploaded file extension and as an exception case by the prefix of header_ for the CSV document with a header. The target Amazon Kendra index is identified by the S3 bucket folder name, which has the index ID as the suffix; for example, faq-1f01abb8-341c-4921-ad16-139ee517a845.

Upload additional FAQ documents

Amazon Kendra FAQ supports three types of file format: CSV, CSV_WITH_HEADER, and JSON. Make sure that when you upload a CSV file with the header, the file name should have a prefix with header_ (this is only when using the CSV file format with a header in its contents). To upload your FAQ documents, complete the following steps:

1. Upload the header_demo.csv file to the same folder.
   
2. Verify that the FAQ is created on the Amazon Kendra console.
   

FAQ creation is case-sensitive to the file format of the FAQ document that you upload. For example, if you upload demo.json and demo.JSON, both are treated as unique objects in Amazon S3. Therefore, this action creates two FAQs, such as demo-json-faq-22-09-2022-20-09-11 and demo-JSON-faq-22-09-2022-20-09-11.

1. Upload demo.JSON.
   
2. Verify that the FAQ for demo.JSON is created on the Amazon Kendra console.
   

Create a new version of the index FAQ

Now the solution is self-sufficient and able to work independently whenever you upload a new version of the FAQ document in Amazon S3.

To test this, upload a new updated version of your demo.json FAQ document to the faq-<index-id> folder. When you navigate to the FAQ for the index, there will be an FAQ named <file-name>-faq-<Date-Time>.

This solution creates a new version of the FAQ for the new version of the FAQ document that was uploaded in Amazon S3. When the FAQ is active, it deletes the older version of the FAQ for the same document.

Create an FAQ with a description

This solution also supports creating an FAQ with a description when files are named in a specific manner: <document_name>-desc-<your faq description>.fileformat[json|csv]. For example, demo-desc-hello world.json. Upload this FAQ document to the faq-<index-id> folder.

After you upload the document, the FAQ will be created and it will have the description as mentioned in the file name.

You should only use -desc- when you must add a description to an FAQ. If you upload a file with the same document_name prefix, it will delete the old FAQ created from the document_name.fileformat FAQ document and create a new FAQ with the description.

Clean up

To clean up, perform the following actions:

1. Empty the S3 bucket that was created by the CloudFormation stack to store the FAQ documents. For instructions, refer to Emptying a bucket.
2. Delete the CloudFormation stack. For instructions, refer to Deleting a stack on the AWS CloudFormation console.

Conclusion

In this post, we introduced an automated way to manage your Amazon Kendra FAQs. After implementing this solution, you should be able to create and delete FAQs just by uploading them to an S3 bucket. This way, you save time by avoiding repetitive manual changes and troubleshooting inconsistent issues that are caused by unexpected operational incidents. You can also audit Amazon Kendra FAQs across your organization with confidence.

Do you have feedback about this post? Submit your comments in the comments section. You can also post questions on the AWS re:Post forum.

About the Author

Debojit is a DevOps consultant who specializes in helping customers deliver secure and reliable solutions using AWS services. He concentrates on infrastructure development and building serverless solutions with AWS and DevOps. Apart from work, Debojit enjoys watching movies and spending time with his family.

Glenn is a Cloud Architect at AWS. He utilizes technology to help customers deliver on their desired outcomes in their cloud adoption journey. His current focus is DevOps and developing open-source software.

Shalabh is a Senior Consultant based in London. His main focus is helping companies deliver secure, reliable, and fast solutions using AWS services. He gets very excited about customers innovating with AWS and DevOps. Outside of work, Shalabh is a cricket fan and a passionate singer.

Read More

Time series are sequences of data points that occur in successive order over some period of time. We often analyze these data points to make better business decisions or gain competitive advantages. An example is Shimamura Music, who used Amazon Forecast to improve shortage rates and increase business efficiency. Another great example is Arneg, who used Forecast to predict maintenance needs.

AWS provides various services catered to time series data that are low code/no code, which both machine learning (ML) and non-ML practitioners can use for building ML solutions. These includes libraries and services like AutoGluon, Amazon SageMaker Canvas, Amazon SageMaker Data Wrangler, Amazon SageMaker Autopilot, and Amazon Forecast.

In this post, we seek to separate a time series dataset into individual clusters that exhibit a higher degree of similarity between its data points and reduce noise. The purpose is to improve accuracy by either training a global model that contains the cluster configuration or have local models specific to each cluster.

We explore how to extract characteristics, also called features, from time series data using the TSFresh library—a Python package for computing a large number of time series characteristics—and perform clustering using the K-Means algorithm implemented in the scikit-learn library.

We use the Time Series Clustering using TSFresh + KMeans notebook, which is available on our GitHub repo. We recommend running this notebook on Amazon SageMaker Studio, a web-based, integrated development environment (IDE) for ML.

Solution overview

Clustering is an unsupervised ML technique that groups items together based on a distance metric. The Euclidean distance is most commonly used for non-sequential datasets. However, because a time series inherently has a sequence (timestamp), the Euclidean distance doesn’t work well when used directly on time series because it’s invariant to time shifts, ignoring the time dimension of data. For a more detailed explanation, refer to Time Series Classification and Clustering with Python. A better distance metric that works directly on time series is Dynamic Time Warping (DTW). For an example of clustering based on this metric, refer to Cluster time series data for use with Amazon Forecast.

In this post, we generate features from the time series dataset using the TSFresh Python library for data extraction. TSFresh is a library that calculates a large number of time series characteristics, which include the standard deviation, quantile, and Fourier entropy, among others. This allows us to remove the time dimensionality of the dataset and apply common techniques that work for data with flattened formats. In addition to TSFresh, we also use StandardScaler, which standardizes features by removing the mean and scaling to unit variance, and Principal component analysis (PCA) to perform dimensionality reduction. Scaling reduces the distance between data points, which in turn promotes stability in the model training process, and dimensionality reduction allows the model to learn from fewer features while retaining the major trends and patterns, thereby enabling more efficient training.

Data loading

For this example, we use the UCI Online Retail II Data Set and perform basic data cleansing and preparation steps as detailed in the Data Cleaning and Preparation notebook.

Feature extraction with TSFresh

Let’s start by using TSFresh to extract features from our time series dataset:

from tsfresh import extract_features
extracted_features = extract_features(
    df_final, 
    column_id="StockCode", 
    column_sort="timestamp")

Note that our data has been converted from a time series to a table comparing StockCode values vs. Feature values.

Next, we drop all features with n/a values by utilizing the dropna method:

extracted_features_cleaned=extracted_features
extracted_features_cleaned=extracted_features_cleaned.dropna(axis=1)

Then we scale the features using StandardScaler. The values in the extracted features consist of both negative and positive values. Therefore, we use StandardScaler instead of MinMaxScaler:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
extracted_features_cleaned_std = scaler.fit_transform(extracted_features_cleaned)

We use PCA to do dimensionality reduction:

from sklearn.decomposition import PCA
pca = PCA()
pca.fit(extracted_features_cleaned_std)

And we determine the optimal number of components for PCA:

plt.figure(figsize=(20,10))
plt.grid()
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of components')
plt.ylabel('cumulative explained variance')

The explained variance ratio is the percentage of variance attributed to each of the selected components. Typically, you determine the number of components to include in your model by cumulatively adding the explained variance ratio of each component until you reach 0.8–0.9 to avoid overfitting. The optimal value usually occurs at the elbow.

As shown in the following chart, the elbow value is approximately 100. Therefore, we use 100 as the number of components for PCA.

Clustering with K-Means

Now let’s use K-Means with the Euclidean distance metric for clustering. In the following code snippet, we determine the optimal number of clusters. Adding more clusters decreases the inertia value, but it also decreases the information contained in each cluster. Additionally, more clusters means more local models to maintain. Therefore, we want to have a small cluster size with a relatively low inertia value. The elbow heuristic works well for finding the optimal number of clusters.

from sklearn.cluster import KMeans
wcss = []
for i in range(1,10):
    km = KMeans(n_clusters=i) 
    km.fit(scores_pca)
    wcss.append(km.inertia_)
plt.figure(figsize=(20,10))
plt.grid()
plt.plot(range(1,10),wcss,marker='o',linestyle='--')
plt.xlabel('number of clusters')
plt.ylabel('WCSSS')

The following chart visualizes our findings.

Based on this chart, we have decided to use two clusters for K-Means. We made this decision because the within-cluster sum of squares (WCSS) decreases at the highest rate between one and two clusters. It’s important to balance ease of maintenance with model performance and complexity, because although WCSS continues to decrease with more clusters, additional clusters increase the risk of overfitting. Furthermore, slight variations in the dataset can unexpectedly reduce accuracy.

It’s important to note that both clustering methods, K-Means with Euclidian distance (discussed in this post) and K-means algorithm with DTW, have their strengths and weaknesses. The best approach depends on the nature of your data and the forecasting methods you’re using. Therefore, we highly recommend experimenting with both approaches and comparing their performance to gain a more holistic understanding of your data.

Conclusion

In this post, we discussed the powerful techniques of feature extraction and clustering for time series data. Specifically, we showed how to use TSFresh, a popular Python library for feature extraction, to preprocess your time series data and obtain meaningful features.

When the clustering step is complete, you can train multiple Forecast models for each cluster, or use the cluster configuration as a feature. Refer to the Amazon Forecast Developer Guide for information about data ingestion, predictor training, and generating forecasts. If you have item metadata and related time series data, you can also include these as input datasets for training in Forecast. For more information, refer to Start your successful journey with time series forecasting with Amazon Forecast.

References

• Dua, D. and Graff, C. (2019). UCI Machine Learning Repository: http://archive.ics.uci.edu/ml. Irvine, CA: University of California, School of Information and Computer Science.
• Direct link to UCI dataset
• Scikit-learn GitHub repo
• TSFresh GitHub repo

About the Authors

Aleksandr Patrushev is AI/ML Specialist Solutions Architect at AWS, based in Luxembourg. He is passionate about the cloud and machine learning, and the way they could change the world. Outside work, he enjoys hiking, sports, and spending time with his family.

Chong En Lim is a Solutions Architect at AWS. He is always exploring ways to help customers innovate and improve their workflows. In his free time, he loves watching anime and listening to music.

Egor Miasnikov is a Solutions Architect at AWS based in Germany. He is passionate about the digital transformation of our lives, businesses, and the world itself, as well as the role of artificial intelligence in this transformation. Outside of work, he enjoys reading adventure books, hiking, and spending time with his family.

Read More