Computer Vision: Digitising Our Lego Schedule Board

To keep things simple, for my initial attempt I chose to focus only on categorising the colour of each block (e.g. orange), rather than working out the project that colour is associated with (which would involve deciphering the legend in the top corner).

For practical reasons, the schedule would be read in from a webcam attached to the ceiling. Key challenges in reading the schedule would be:

1. Correcting for the perspective of the webcam, to make detecting blocks easier.
2. Finding each of the boards, and overlaying a grid so we can identify each of the bricks within it.
3. Distinguishing between differently coloured blocks, some of which are very similarly coloured.

To process the images of the schedule board, I used a piece of computer vision software from MATLAB, which makes prototyping computer vision workflows really simple.

As can be seen from the image, all the boards are light grey, and the backdrop is largely green. This makes the task of separating the bit we are interested in somewhat easier.

For this reason, as an initial step, we process the raw image so that all the suitably grey pixels are highlighted. This step simplifies things somewhat, as it means we can remove irrelevant (at this stage) colour information, and focus on a simple binary image of white and black, where white represents a greyish pixel.

Afterwards, we can see that this step has largely picked out the boards. However, there are a few imperfections. Clearly, some things that are not Lego boards have been picked out, such as the silver buckets on the right-hand side of the bucket. Also, even though Lego’s diligent production processes have ensured the boards are uniformly grey, not all the pixels that “belong” to the boards have been picked out. This is due to differences in shading and illumination and is particularly apparent on the bottom-most board.

To correct these irregularities, I performed a binary image operation called morphological opening, which essentially spreads each white pixel a small distance into its neighbours to form a new, larger, region of white pixels — with holes and imperfections removed — before then eroding the entirety of this newly formed region so that it is contained within its original bounds — but with internal imperfections still removed. This results in an image like the one below:

Now, the area of each connected binary region is measured, and the four largest ones (that hopefully correspond to boards) are kept, and all others removed. Finally, we fill in any irregularities around the edges using some more morphological operations, to end up with the final binary image identifying the location of the boards.

It turns out, that because this process is based on simple assumptions (the boards are grey, the boards dominate the image, the background is not grey) it is also very robust, and works in a large variety of camera angles and lighting conditions. Robustness is desirable as it means we avoid having to recalibrate the detector every time we move the camera, or turn on the lights!

Next, we detect the dominant straight lines within this edge image, using a Hough transform, which is a well-established way of finding simple shapes inside binary images. All lines shorter than a certain length are removed, and then all lines are removed that are not the furthest up, down, left, or right. This leaves four lines, two horizontal and two vertical, and the corners of the boards can be found by calculating the intersection points of these lines using standard formulae. The lines detected for the edge image above are shown below:

To register the image so it is head-on, it is then intuitively a case of “stretching” out the above image so that the green lines are vertical, and the red lines are horizontal. In practice this involves calculating a transformation matrix using the four corner points we calculated. This gives a registered image like the following:

You can see that the top left and top right regions are now black, as these regions were not captured in the original image. Detecting the brick colours is now just a case of overlaying a uniform grid over each board and classifying the colour inside.

By manually labelling the bricks in a few example boards, we can establish a “ground truth” that can be used in a supervised learning setting to train a classifier based on each block’s pixel values. The classifier can then be used to predict the colour value of new, unseen bricks! Using machine learning in this way approximately halves the error rate from naively classifying each block based on simple colour distance. An example of a resulting classification is below:

There are a few errors, but on the whole, it does an ok job. It seems to get confused by grey-ish coloured bricks, and shadows and the underlying cause of these errors weren’t easily fixable in the time I had remaining on the project, so I decided to stop development here and come back to improve reliability in the future.