06.2 Cognitive Services (Custom Vision)

Introduction

In the Computer Vision scenario, you learnt how to use a generic engine for classifying images. Now, you will learn how to train a custom model (similar to the Custom Model scenario) using Custom Vision Services, that allows to leverage Cognitive Services infrastructure for training instead of using on-premises infrastructure.

Setting up Cognitive Services - Custom Vision

Create new Custom Vision Classification project

Following steps are extracted from the guide How to build a classifier with Custom Vision; if you need more detail, visit Custom Vision documentation.

Before starting, you need a valid Custom Vision account. If you haven't got it already, please go to Custom Vision, click in the Sign In button in the page and follow the process to sign in using your Microsoft Live account:

Next, you need to create a new classifier project: click on New Project. The required fields are name of the project (for example, eShopOnContainersAI) and project type (Classification) and Domain (Retail (compact) - the compact domain type allows you to download the model later), as shown in the following screenshot:

Uploading products

The following step will be to upload model training samples (product images). You can use same image dataset used in the Custom Model. Click in the Add images button, next in Browse local files button and locate a folder inside your repository named workbench / data / train. This folder contains images from five different products, which will be used to train your custom model.

Inside the folder named frisbee, select all files and click the Open button; this will upload the selected images to the Custom Vision service. You can review previous steps in the following screenshot:

Once the files are uploaded, you need to assign them the correct tag; type 'frisbee' in the input textbox, then click on the + button and finally hit the Upload button as described in the following screenshot:

When the tagging process finishes, you should get an screen similar to the following one:

You need to repeat previous process and upload each product in the train folder. In order to upload the following batch of images, you should click in the first button besides the Refinetab, as shown in the figure:

After uploading all images, you should have 5 tags (product categories): bracelet, earrings, frisbee, parasol, thermometer . You can filter tags checking the tag; additionally, you might click on the untagged button and verify there is no image shown (so all uploaded images should be tagged)

Training and model deployment

After uploading and tagging all images, you can start training your custom model. Click on the green button at the top of the screen, and the training will start as shown in the following screenshot:

When the training finishes, you should get an screen similar to following one:

The Precision and Recall metrics are generally used to evaluate the classification model's goodness.

One last step: you must set the training (iteration) as default. Depending on how many trainings you make, you should always define your "default" iteration. In order to do so, select the Performance panel, then click on the desired Iteration in the left panel, and finally click on the "checked" button to mark current iteration as default, as shown below:

Setting up Visual Studio project

Custom Vision Services allows you to work in two ways: using the model that is hosted in Azure (online model) and using the same model in your own-premises (offline). You need two different settings to set up Custom Vision in eShopOnContainersAI

Common steps

eShopOnContainersAI uses a configuration flag to select the search product by image model. The flag is named ESHOPAI_PRODUCTSEARCHIMAGEBASED_APPROACH, and is persisted in the .env file:

#ESHOPAI_PRODUCTSEARCHIMAGEBASED_APPROACH=<TensorFlowPreTrained|TensorFlowCustom|ComputerVision|CustomVisionOnline|CustomVisionOffline>
ESHOPAI_PRODUCTSEARCHIMAGEBASED_APPROACH=CustomVisionOffline

Online model

You need the Prediction Key and the Project Id in order to make prediction requests to the Custom Vision Service. These data is available in the Custom Vision portal, click in the Settings button, and copy the required data, as shown in the picture:

You need to input this data either as environment variable or as configuration in the .env file:

ESHOPAI_CV_PRED_KEY=<YourAzureCustomVisionPredictionKey>
ESHOPAI_CV_PROJECT_ID=<YourAzureCustomVisionProjectId>

Offline model

The same model trained in the cloud might be downloaded and used it offline. Custom Vision Services allows to download the model in several formats (depending on the final environment). In the case of the eShopOnContainersAI microservice, you need to download the model as a Tensorflow model.

In the Custom Vision portal, select the Performance tab, and then, click on the download button, as shown in the next figure:

In the right blade, you need to select the TensorFlow model, and then click in the Download button

The downloaded file is a zip file, containing two files: the frozen model in proto buffers format, and the labels file. You need to unzip this file, and rename the file with the extension .pb to model.pb. Finally, you need to copy the model.pb and labels.txt files to the ModelsAI folder in the AI.ProductSearchImageBased.AzureCognitiveServices.API project, as shown below:

Step 3: Run and test a "Product Search Image-Based" with Custom Vision

When the webmvc application is running, open a local browser at http://localhost:5100 . Then, click on the Choose File button and select an image from your local drive.

In the above figure, the product search image approach is highlighted (CUSTOMVISIONOFFLINE), and shows the result searching for a bracelet image.

Step 4: C# code walkthrough

Custom Vision is hosted in the ai.productsearchimagebased.azurecognitiveservices.api microservice. This microservice uses several environment variables in order to set up keys used by the online model and the approach (strategy) to follow for classifying images (ComputerVision | CustomVisionOnline | CustomVisionOffline)

docker-compose.override.yml

services:
  ai.productsearchimagebased.azurecognitiveservices.api:
    environment:
      ComputerVision__VisionAPIKey: ${ESHOPAI_CS_VIS_KEY}
      ComputerVision__VisionUri: ${ESHOPAI_CS_VIS_URI}
      CustomVision__PredictionKey: ${ESHOPAI_CV_PRED_KEY}
      CustomVision__ProjectId: ${ESHOPAI_CV_PROJECT_ID}
      CognitiveServicesPredictionDefaultModel: ${ESHOPAI_PRODUCTSEARCHIMAGEBASED_APPROACH}

The online model is very easy to use, thanks to the official nuget package (Microsoft.Cognitive.CustomVision.Prediction). After setting up the prediction key and the project id, you can use the method PredictImageWithNoStoreAsync (you can explore the API and discover other useful methods). The scores are filtered using a custom threshold.

CustomVisionClient.cs

var endpoint = new PredictionEndpoint() { ApiKey = settings.CustomVisionPredictionKey };

Cognitive.CustomVision.Prediction.Models.ImagePredictionResultModel result;

using (var testImage = new MemoryStream(image))
{
    result = await endpoint.PredictImageWithNoStoreAsync(settings.CustomVisionProjectId, testImage);
}

return result.Predictions.Select(t => new LabelConfidence() { Label = t.Tag, Probability = (float)t.Probability })
                .Where(c => c.Probability >= settings.Threshold)
                .OrderByDescending(c => c.Probability);

The offline model uses TensorFlowSharp. The image needs to be pre-processed (this processing is done inside the method LoadImage) before using it as input of the deep neural network; after evaluating (classifying) the image, the results are filtered out using a base threshold.

CustomVisionOfflinePrediction.cs

var (model, labels) = LoadModelAndLabels(settings.ModelFilename, settings.LabelsFilename);
var imageTensor = LoadImage(image);

var result = Eval(model, imageTensor, settings.InputTensorName, settings.OutputTensorName, labels).ToArray();

IEnumerable<LabelConfidence> labelsToReturn = result                            
                        .Where(c => c.Probability >= settings.Threshold)
                        .OrderByDescending(c => c.Probability);
return labelsToReturn;