Counting Solution with SageMaker Object Detection Built-in Algorithm

Takahiro Iwasa

Jul 18, 2023

8 min read

Ground Truth Object Detection SageMaker

Object detection algorithms can be effective for counting objects, and SageMaker has some built-in algorithms. This post describes how to use it through the following steps:

• Labeling images using Ground Truth
• Training and deploying your models
• Performing inference

In my personal opinion, users should not use SageMaker inference endpoints directly for production workloads. This post uses it for testing only.

Ground Truth offers the labeling feature.

docs.aws.amazon.com

Use Amazon SageMaker Ground Truth to Label Data - Amazon SageMaker

To train a machine learning model, you need a large, high-quality, labeled dataset. Ground Truth helps you build high-quality training datasets for your machine learning models. With Ground Truth, you can use workers from either Amazon Mechanical Turk, a vendor company that you choose, or an internal, private workforce along with machine learning to enable you to create a labeled dataset. You can use the labeled dataset output from Ground Truth to train your own models. You can also use the output as a training dataset for an Amazon SageMaker model.

Create your labeling job by following the steps below.

First of all, create your labeling workforces. A private team will be created in this post. Team members can be also authenticated with either Cognito or OIDC.

An invitation email has been sent to workers, which displays the labeling portal URL. They can sign up and sign in to the labeling portal using that URL. You can also find the URL, navigating to Private workforce summary > Labeling portal sign-in URL on the management console.

Workers need to sign up and sign in to the labeling portal using information provided in the invitation email.

Hi,

You are invited by [email protected] from <COMPANY> to work on a labeling project

Click on the link below to log into your labeling project.
"https://<LABELING_PORTAL_URL>"

You will need the following username and temporary password provided below to login for the first time.
User name: <USER_NAME>
Temporary password: <PASSWORD>

Once you log in with your temporary password, you will be required to create a new password for your account.
After creating a new password, you can log into your private team to access your labeling project.

If you have any questions, please contact us at [email protected].

After accessing the URL, enter the username and password in the invitation email.

Change the temporal password.

You will be redirected to the top page. On this page, assigned labeling jobs are listed if available.

Navigate switch back to SageMaker management console, start creating labeling job and enter the field values according to the image below. After creating, the labeling job cannot be deleted, so it is better to use the unique value such as one generated uuidgen | tr "[:upper:]" "[:lower:]". Please make sure to click Complete data setup.

When dealing with complex labeling tasks, it may be better to specify a longer value for Task timeout.

After signing in to the labeling portal, you should see the labeling job that has been created above. Click the Start working button.

According to the job instructions, label the dataset. Here is the example.

Once all workers have completed the labeling job, stop the job.

After stopping the labeling job, its final output has been generated in the specified S3 bucket. For the object detection tasks, manifests/output/output.manifest is important.

docs.aws.amazon.com

Output Data - Amazon SageMaker

The output from a labeling job is placed in the Amazon S3 location that you specified in the console or in the call to the CreateLabelingJob operation. Output data appears in this location when the workers have submitted one or more tasks, or when tasks expire. Note that it may take a few minutes for output data to appear in Amazon S3 after the worker submits the task or the task expires.

<YOUR_OUTPUT_PATH>/
|-- annotation-tool/
|-- annotations/
|   |-- consolidated-annotation/
|   `-- worker-response/
|-- manifests/
|   |-- intermediate/
|   `-- output/
|       `-- output.manifest
`-- temp/

Ground Truth generates labeling results by Augmented Manifest format. Please refer to the official documentation for more details.

docs.aws.amazon.com

Output Data - Amazon SageMaker

The output from a labeling job is placed in the Amazon S3 location that you specified in the console or in the call to the CreateLabelingJob operation. Output data appears in this location when the workers have submitted one or more tasks, or when tasks expire. Note that it may take a few minutes for output data to appear in Amazon S3 after the worker submits the task or the task expires.

Once the labeling has been completed, train your model using SageMaker console.

The important fields are the following:

• Job settings
  • Job name
    • The training job cannot be deleted after creation, so it is better to use the unique value such as one generated uuidgen | tr "[:upper:]" "[:lower:]".
  • Algorithm options
    • Algorithm source: Amazon SageMaker built-in algorithm
    • Choose an algorithm
      • Vision - Object Detection (MXNet)
      • Input mode: Pipe
  • Resource configuration
    • Instance type
      • ml.p2.xlarge in this post
      • Only GPU instances support object detection algorithms.
• Hyperparameters
  • num_classes: 1 in this post
  • num_training_samples: Count equal to the manifest lines
• Input data configuration
  • train
    • Channel name: train
    • Input mode: Pipe
    • Content type: application/x-recordio
    • Record wrapper: RecordIO
    • Data source
      • S3
        • S3 data type: AugmentedManifestFile
        • S3 data distribution type: FullyReplicated
        • AugmentedManifestFile attribute names
          • source-ref
          • Key name containing bounding boxes data
        • S3 location: S3 URI of your augmented manifest file for training
  • validation
    • Channel name: validation
• Output data configuration
  • S3 output path: S3 URI of your model artifacts which SageMaker produces

Thanks to the augmented manifest file generated by Ground Truth, we can use Pipe input mode and RecordIO.

docs.aws.amazon.com

Object Detection - MXNet - Amazon SageMaker

The Amazon SageMaker Object Detection - MXNet algorithm identifies object instances in an image.

The augmented manifest format enables you to do training in pipe mode using image files without needing to create RecordIO files.

When using Object Detection with Augmented Manifest, the value of parameter RecordWrapperType must be set as “RecordIO”.

Click Create model to create a model from the training job.

Click Create endpoint to deploy the model from the model page. If you can tolerate cold startup, serverless endpoint should be a better cost-effective choice.

docs.aws.amazon.com

Serverless Inference - Amazon SageMaker

Amazon SageMaker Serverless Inference enables you to deploy your ML model without selecting instance types. Serverless Inference manages the heavy lifting of managing servers so you can focus on leveraging ML.

Serverless Inference is a cost-effective option if you have an infrequent or unpredictable traffic pattern. During times when there are no requests, Serverless Inference scales your endpoint down to 0, helping you to minimize your costs.

Find the SageMaker inference endpoint in your endpoint detail page. The endpoint can be directly accessed using tools such as curl, Postman, and others.

docs.aws.amazon.com

InvokeEndpoint - Amazon SageMaker

After you deploy a model into production using Amazon SageMaker hosting services, your client applications use this API to get inferences from the model hosted at the specified endpoint.

An example image below is using Postman. Specify the following information to provide an AWS Signature V4 header.

docs.aws.amazon.com

Authenticating Requests (AWS Signature Version 4) - Amazon Simple Storage Service

Use access keys to create a signing key and authenticate your request using AWS Signature Version 4.

• AccessKey
• SecretKey
• Session Token: I strongly recommend using a temporary credential instead of a permanent one.
• AWS Region: Region in which your SageMaker inference endpoint has been started
• Service Name: sagemaker

Include Accept: application/json header.

Since the trained model expects binary data as input, provide your image in binary format.

You can also use boto3 invoke_endpoint API to perform inference.

docs.aws.amazon.com

Invoke a serverless endpoint - Amazon SageMaker

To invoke a serverless endpoint, you must make an HTTP request to the endpoint.

import json

import boto3

runtime = boto3.client('sagemaker-runtime')

endpoint_name = '<YOUR_ENDPOINT_NAME>'
content_type = 'application/x-image'
payload = None

# Read an image
with open('/path/to/image.jpg', 'rb') as f:
    payload = f.read()

# Run an inference
response = runtime.invoke_endpoint(
    EndpointName=endpoint_name,
    ContentType=content_type,
    Body=payload
)

# The type of response['Body'] is botocore.response.StreamingBody
# See https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html
body = response['Body'].read()

# Print the result
predictions = json.loads(body.decode())
predictions = json.dumps(predictions, indent=2)
print(predictions)

# Write the result to a file
with open('./response.json', 'w') as f:
    f.write(predictions)

The inference response is in JSON format, including class label IDs, confidence scores, and bounding box coordinates. Please keep it in mind that the bounding box coordinates are relative to the actual image size.

Fore more information about the format, please refer to the official documentation.

docs.aws.amazon.com

Object Detection Request and Response Formats - Amazon SageMaker

Query a trained model by using the model's endpoint. The endpoint takes .jpg and .png image formats with image/jpeg and image/png content-types. The response is the class index with a confidence score and bounding box coordinates for all objects within the image encoded in JSON format. The following is an example of response .json file:

{
  "prediction": [
    [
      0.0,
      0.9953756332397461,
      0.3821756839752197,
      0.007661208510398865,
      0.525381863117218,
      0.19436971843242645
    ],
    [
      0.0,
      0.9928023219108582,
      0.3435703217983246,
      0.23781903088092804,
      0.5533013343811035,
      0.6385164260864258
    ],
    [
      0.0,
      0.9911478757858276,
      0.15510153770446777,
...
      0.9990172982215881
    ]
  ]
}

To visualize the inference response, this post uses Jupyter Notebook and matplotlib.

import json

import matplotlib.patches as patches
import matplotlib.pyplot as plt
from PIL import Image

# Configure plot
plt.figure()
axes = plt.axes()

# Read an image
im = Image.open('/path/to/image.jpg')
# Display the image
plt.imshow(im)

# Read SageMaker inference predictions
with open('response.json') as f:
    predictions = json.loads(f.read())['prediction']

# Set initial count
count = 0

# Create rectangles
for prediction in predictions:
    score = prediction[1]
    if score < 0.2:
        continue

    # Count up
    count += 1

    x = prediction[2] * im.width
    y = prediction[3] * im.height
    width = prediction[4] * im.width - x
    height = prediction[5] * im.height - y

    rect = patches.Rectangle((x, y), width, height, linewidth=1, edgecolor='r', facecolor='none')
    axes.annotate(count, (x + width / 2, y + height / 2), color='yellow', weight='bold', fontsize=18, ha='center', va='center')
    axes.add_patch(rect)

# Display the rectangles
plt.show()

Machine Learning using Object Detection is useful for counting things, and SageMaker provides the necessary ecosystem.

I hope you will find this post useful.