Training the Machine: Labeling Images for Deep Learning
As part of my role with alooki, I'm working to automate the counting of blue recycling bags captured in video recorded by a tiny computer - the Raspberry Pi. In order to do that, I'm training a machine learning framework (TensorFlow) to detect blue bags in those videos. This post walks you through what I've found to be the most time consuming part of the process: image labeling.
Overview
Before you can train a machine to detect pictures of objects, you need to have lots of pictures of the object(s) you want to detect along with labels for those objects. In other words, you need to manually review the pictures and identify the portions of those pictures that you want to have the computer learn to recognize. This is particularly true if the objects appear in a wide variety of settings with lots of different backgrounds.
My ultimate goal is to train Google’s TensorFlow to detect blue bag recyclables in various pictures I’ve extracted from video, hence the reason I need a lot of labeled pictures. Before we dig in, I’m assuming you already have a set of pictures ready to go. If you don’t, I’ll walk you through how to gather pics or create your own from video in my next blog post.
The high-level image labeling process looks like this:
Install image labeling program
Label your first image
Lather. Rinse. Repeat. (Helpful Hints)
1. Install Image Labeling Program
I use the labelImage software to label the objects in my images. Installation is available for Windows and Linux. Installation on a Mac is unfortunately more complicated, so I’m not covering that here. More technical details than you probably ever want to know are available on GitHub.
Windows installation
Make sure you’ve downloaded the installation file (windows_v1.4.0.zip).
Unzip the windows_v1.4.0.zip file into the desired directory. This creates a windows_v1.4.0 folder containing the labelImg.exe program.
Launch the labelImg.exe program . The image labeling window appears.
Linux installation
Make sure you’ve downloaded the installation file (linux_v1.4.0.zip).
Open a terminal/command line window and extract the zip file into the desired folder with the command: unzip linux_v1.4.0.zip
This creates a linux_v1.4.0 folder containing the ‘labelImg’ program.
Launch the program with the command: ./labelImg
The image labeling window appears.
2. Label Your First Image
Using your shiny new labelImage program, let’s start labeling images.
1. The first thing you need to do is open the folder containing your pictures by clicking “Open Dir” and selecting your image folder. As a side note, since I’ll be using these labeled images with Google's deep learning framework (TensorFlow), all of my images are in JPEG format.
2. Click “Open” and the first image in your image directory is displayed. See the screen shot below for an example of my first image being shown in the labeling window.
3. Press the ‘w’ key to being labeling the image. You’ll notice that your mouse pointer changes to a “+” to indicate that you are in selection mode.
4. Move your mouse near the object of interest, left click and drag until the rectangle selection area covers the object you're labeling.
5. Release the left mouse button to display the label dialog box . The label dialog box is where you assign the label for the object you just selected. Add new labels by typing into the space provided (garbagebin in my example). Click “OK” to assign the label.
You can assign a previously existing label by double clicking the appropriate label from the list of labels shown in the label dialog box. 6. You can specify multiple objects in the same image by repeating the steps above: press ‘w’, drag the mouse to select, assign a label. See below for an example. Notice the green rectangles around the two blue bags and the garbage bin in the bottom right of the picture. Also notice the labels that I’ve assigned are shown on the right.
7. Once you’ve finished highlighting and labeling the objects in your picture, click “Save” to save the label information. By default, the label info for your picture gets saved to an XML file in the same folder as your pictures.
8. Click “Next Image” to display the next image in the directory.
3. Lather. Rinse. Repeat. (Helpful Hints)
There’s no easy alternative to the mind-numbing repetitive labeling of many, many images to train a reasonably accurate object detector. The number varies based on the variation of the images and objects but hundreds to thousands are common numbers I’ve come across.
It quickly gets repetitious. Seriously repetitious. So much so that I found myself looking for shortcuts to make it go at least a little bit faster. Below are the shortcut keys I found most helpful:
‘W’ - enable selection
CTRL+S - save label data for the current pic
'D' - load next image
'A' - load previous image
I also found that I could resize or delete selections I had made by mistake which comes in handy when your mind starts to drift...
Lastly, since I was labeling multiple different objects, I noticed that the last label I had chosen was the default for the next object I selected. It saved a few clicks to highlight the same type of object one after the other because of this default label functionality.
That’s it. Nothing left to do but dig in - happy labeling!