How to Train the AI Face Recognition Engine
Zoom allows you to train the face recognition engine to recognize one or more faces with a custom label that represents a person’s name. Once you finish training, you can then use Zoom’s AI Hub to recognize all the faces in images that match the trained faces. After that, you can search by person’s name across millions of images and find matching faces in these images.
In this article, we cover how to train our face recognition engine. For each person that you wish to recognize, you will need to create 5-10 cropped images for each age group for a person.
The process
Before you begin
You should have access to a library of images that represent a person. You should be able to use the Zoom photo cropping tool or have access to a photo editing software such as Adobe Photoshop.
Prerequisites
- Your media manager has configured the Zoom AI Hub to pick up your unrecognized images and run AI Face Recognition using the training set you will create.
- You must have access to the Zoom desktop apps
- You must be in a projectadmin role, who can execute AI face training jobs using the Zoom Asset Browser app on the desktop.
- You must be prepared to iterate over the training process a few times with a set of test images till you are happy with the recognition.
- Follow the guidelines below for creating the training set.
Benefits
The Zoom face Training process allows you to:
- Create a grouping of cropped faces to be used as a training set.
- Once you add sufficient cropped images for a person, you can let the AI recognize the person from any number of images you have in the Zoom repository.
Procedure
Step 1: Create a folder structure for training set
We require arranging the cropped images in age specific folder structure in Zoom. Create a project vault or a subfolder under an existing Zoom Project vault called “AI Training” that will hold the cropped images you will use for training. The default sub-folder location where the training set should reside is: /AI Training/Face Dataset/<Person Name>/<Age Range>. For example:
Step 2: Name your crop angle
Under the age based sub folder, add your cropped images (PNG or JPEG) with meaningful names or use a Zoom custom metadata field to track the shot angle. For example:
Step 3: Label the faces
For each of the cropped images per the guidelines below, after ingesting them into the Zoom repository, please attach a label using a metadata field: AI CONTROL::Face Label
Step 4: Submit the Training Set
Once you have completed creating the training set, submit it to the Zoom AI hub using the Asset Browser › Create Job › AI Face Training as following:
- Go to the AI Training project in Asset Browser Browser View.
- Right click on the AI Training/Face Dataset folder.
- Select Create Job.
- Select AI Face Training as Template.
- Select AI Face Training as Form.
- Click on the button Create.
This will submit the AI Face Training Job to the Zoom AI hub.
When to re-submit a training job?
You should re-submit the AI face training job in following scenarios:
- If a new face is added to the training dataset under the /AI Training/Face Dataset/ folder.
- If you edit a person’s face name in the Face Label metadata field.
- If you delete a face from a training dataset.
Step 5: Check the Face Training Status
Once an AI Face Training job is submitted, you will receive various emails about your job status.
You will be notified by email:
- When the training job starts.
- When the training job completes.
- When the training job fails.
In case of errors, you can view the errors in the email attachment. Please resolve the errors before submitting the training job again.
Step : Perform Face Recognition
Once training is completed, you are ready to submit your images for AI based face recognition.
Sign-in to Zoom Asset Browser or the Desktop/Web Client. Select the images on which you want to run the Face Recognition on.
Click on Edit Metadata in Zoom Desktop/Web Client or use the Asset Browser to edit the metadata. Set the AI CONTROL:Face Recognition metadata to Submitted.
Setting the Face Recognition metadata to Submitted will trigger the AI recognition job on the server. This has financial implications especially if you submit large image batches.
Once the face recognition job has started, this metadata will change to Running. And once the job is done, this metadata will change to Finished or Failed. Face recognition jobs can fail if the AI Hub configuration is invalid.
To find out why Face Recognition jobs have failed, you need to use the Web based AI Hub dashboard and search for the Asset ID for which the job has failed.
Once Face recognition is finished, you should be able to open the asset and see the detected faces and person names in the Zoom Web Client or the Zoom Desktop app.
Step 7: Review the Results
Please review the images on which the AI CONTROL:Face Recognition metadata is set to Finished in the Zoom Desktop or the Web Client.
There is no email alert right now that notifies you when the recognition is finished on all your images. Easiest way is to use the Advanced Search in Zoom Desktop or Asset Browser to locate the finished assets.
If you are the media manager with superadmin access you can monitor the AI Hub dashboard for completion.
If you made a mistake in labeling a face please see the instructions below on how to fix the typos.
Spelling Mistakes & Typos in Face Labels
Spelling mistakes
A person’s name, Mike Chang, for example is misspelled as Mike Cheng in the metadata while performing the training steps above. The best way to avoid this is to use a drop down menu list in the metadata field, when selecting a name instead of a free text field. But mistakes happen.
How to fix spelling mistakes?
If you have made a spelling mistake in the person’s name:
- Visit the problematic training dataset asset in Zoom, where you have applied the wrong Face Label.
- Edit the Face Label metadata property and correct the name. For Example, edit the metadata Mike Cheng in the asset Mike Chang Right Side Face.jpeg to Face Label, Mike Chang.
- Re-submit a new training job using the Zoom Asset Browser.
- Wait for the Training to finish.
This will fix the problem for future face recognitions, but it won’t fix the typo in already recognized assets.
To fix the previously recognized images for spelling mistakes:
- Goto to the AI Training vault in Zoom Asset Browser.
Note: You have to be in the AI Training project vault, otherwise you will not get the menu option to run the Rename AI Name job. - Right click on any asset or folder.
- Select Create Job.
- Select Template: Rename AI Name
- Select Form: Rename AI Name
- Enter the Old Name. Enter the New Name.
- Click on the Create button
- Wait for at-least 15 mins.
Check the previously recognized assets which had the spelling mistakes Note: Using this action you can only fix the typos made in the name of the person. You cannot re-label a person to a completely different person's label already present in the training set.
Mislabelled Faces
Consider a scenario, where there are two distinct persons in the training data set:
- Person 1: Mike Chang
- Person 2: Linda Wafer
If Mike Chang’s face crop images are mislabeled as Linda Wafer that would lead to Mike’s face being recognized as Linda. Not good.
How to fix mislabelled faces?
If you have mislabelled a person’s name:
- Visit the problematic training dataset asset in the Zoom Asset Browser.
- Correct the Face Label metadata property and change it to the new name. For example: edit the metadata value of Mike Chang Face.jpeg, from Face Label=Linda Wafer to Face Label=Mike Chang.
- Submit a new training job as opposed to a rename job.
- Wait for the Training to finish. After the re-training, any new face recognitions will pick up the correct name.
This will fix the problem for future face recognitions, but it won’t fix the mislabelled faces in the already recognized assets. To fix that:
- Search & find the mislabeled assets in the Zoom Asset Browser.
- Reset the AI CONTROL:Face Recognition metadata on the selected assets to Submitted from the current value Finished.
This will resubmit these images to the AI Face Recognition engine and incur the face recognition costs again. It's best to avoid this by manually validating the labels on the cropped images before recognition is started.
Guidelines for Training set
As mentioned above, the training set contains the cropped images of faces for one or more persons that you will use to train the AI Face Recognition engine to recognize faces and tag them with your custom labels.
General face cropping guidelines
- Prefer cropped images from professional photo shoots if available.
- Minimum cropped face dimension must be 50×50 pixels. Higher would be better.
- Use a face that occupies a large proportion of the crop. Training set images where the face occupies a larger proportion of the image lead to greater accuracy during the recognition phase.
- Cropped face should include a full head, hair, ear, neck and some part of the shoulders.
- Avoid faces with hats, glasses, headbands, and masks.
- Avoid obscure or concealed faces.
- Make sure at least 70% percent of the face is visible in the cropped image.
- Avoid faces with partial shadows, prefer faces with flat lighting.
- A cropped image must contain only a single person’s face.
- To cover various life stages from childhood to adulthood, include crops from these age brackets (years): 0-1, 1-2, 3-5, 5-10, thereafter every decade such as 11-20, 21-30 and so on.
- Include between 5 to 10 cropped images per age group
Face angle guidelines
You will need at-least 5 crops with varying face angles: yaw, pitch & straight-on. Please ensure you include at least 1 cropped image from the below list of 5 face angles:
Face Straight on Crop
- face crop straight on
Face Pitch Crop
- face tilted down with a pitch of 30 degrees or less
- face tilted up with a pitch of 45 degrees or less
Face Yaw Crop
- face turned left with a yaw of 45 degrees or less.
- face turned right with a yaw of 45 degrees or less
After adding at-least one set of images with above angles, try to add a few images of the person in an age group with a beard or braces if available.
Examples of Bad Crops
Examples of Good Crops
Training set gotchas
- Detected face name cannot be renamed. Make sure the metadata field has the correct person’s name before you start the training. Otherwise the images will need to be submitted again for training.
- Removing a face from a training set after it has been used for face recognition, will not remove the metadata for earlier recognised images.
- To clear out AI metadata for recognized faces, just re-run the AI Face recognition job on the selected assets or folders. This will delete previous face recognition metadata and insert new metadata based on the recognition job
- Keep in mind the cost of AI is proportional to both the number of images as well as how many iterations of AI Face recognition your perform
Next Steps
Setup your Project & Media Store Folders
Before you start editing projects using the Zoom Desktop app or Plugins, you need to setup your Zoom workspace folders. This article will walk you through the setup along with common tips & tricks.
- Last modified on