Developing an Object Detection App for the Blind

As a developer working in an agency that collaborates with charities all over the world, it’s inevitable that at some point I’d be influenced by these amazing organisations.

Seeing first-hand the impact that these charities are having has made me not only want to grow as a developer but as a person more so.

The last time I was inspired to be a better person?

Working on a project with Vista. This project has been one of my favourites to date, an incredible charity with an incredible cause that does Tech For Good a favour and has a genuine passion for improving the lives of their community.

SIMI (pronounced See-Mee) aspires to be a tool that makes the daily lives of those living with visual impairments a little easier.

Here are the elements that allow him to do so…

Arguably the most important feature is SIMI’s ability to help in times of crisis. If someone who is visually impaired has an accident or emergency, (like getting hurt/lost/overwhelmed) they will be able to ask SIMI to make a call for them.

The ‘First Contact’ feature allows you to add emergency contacts in case of (you guessed it) an emergency. This means that you can quickly call or notify these people when time is of the essence by telling SIMI to ‘Call Dad’ (for example).

SIMI’s object recognition means that he can not only recognise what/who is in front of you but can even recognise a familiar face. Meaning SIMI could be like “Hey James” and you’d know exactly who it was approaching you.

The ‘What I see’ feature allows SIMI users to send a picture of what the app is seeing to their ‘First Contacts’. This can help users get a second opinion on the apps’ recognition capability or simply share a picture of something nice with your loved ones.

This could even go as far as to warn you when you’re about to step out onto a busy road. Or let you know when it’s safe to cross.

Ask SIMI questions like “what’s the weather like?” and it’s API will collect data on your query and feedback with the answer.

With the inclusion of GPS features, SIMI is also able to tell you things like your location and guide you to your desired location (these could be places like your bathroom or the local shop).

Another cool feature is that SIMI can scan the barcodes of items it recognises, like books for example, and search the internet to find the product and offer channels to purchase it.

Taking this forward I want SIMI to be able to actually read the books too, another way of improving the lives of those with visual impairments.

Since I wanted to help as many people as possible within this community, it was crucial to avoid limitations to the project, such as storage space, RAM, camera quality and versions of Android…

The app was designed to work on Android Lollipop (v21), but with a little modification, could also work on Android KitKat (v19). Which translates to support 80.7% and 93.6% of the Android user population.

If you’re familiar with object detection, then you have undoubtedly heard of TensorFlow (a graph computation library).

With the vast majority of developers using this software to construct deep neural networks.

This and OpenVC (an open-source computer vision and machine learning library).

These two libraries are best suited for object detection, and another benefit is that they can be used separately or together.

Baring in mind I have limited experience in object detection, I found that the configuration was easiest with TensorFlow, plus we found pre-trained models for object detection.

My first attempt was to use the default Android speech recogniser. But I found that it doesn’t support continuous speech recognition, and on top of this, you have to have internet access (unless targeting Android Marshmallow version or newer).

Moreover, the recogniser is not really configurable (you can’t set a keyword – ‘Hello Google‘).

DroidSpeech (unlike the default recogniser) supports continuous speech detection (plus its open-source software), but at the same time also requires internet access and does not support keyword searches.

PocketSphinx is a lightweight speech recognition engine, specifically tuned for handheld and mobile devices (although it’s just as effective on desktop). The library supports continuous speech detection and doesn’t require internet access.

But the best thing about the library is the fact that it supports keyword and phrase searches.

In the app’s first version, SIMI only supported the Pocketsphinx library, which is good at detecting keywords but not great at understanding full sentences.

In the end, I implemented SpeechRecogniser, a library that I’d actually rejected in the early stages of development. Now, the app can switch between the two libraries to make sense of what you’re saying.

One thing that has become apparent since starting my Innovation Time is that I have made far too many assumptions. As a prototype, SIMI worked well in recognising speech, detecting objects and interacting with the user.

However, the problems SIMI solves are quite obvious/simple ones. To take the mobile app development project forward, the first thing I’d want to do is sit down with members of this community and get them to discuss the problems they are facing…