Microsoft has painted a picture of a future where you interact with websites, mobile apps, and everything else by speaking to chat bots who can understand you using natural language thanks to machine learning.

But while you may not be sold on the need to talk to a virtual concierge to book your next hotel stay or restaurant reservation, the company also used today’s BUILD conference keynote to show off something else that’s possible thanks to machine learning: Seeing AI is an app that can use a smartphone or smart glass product to describe what’s going on in the world to a visually impaired user.


Microsoft software engineer Saqib Shaikh has been blind since he was 7 years old. But a mobile app allows him to tap a button on a pair of PivotHead smart glasses to snap a photo and have the device recognize visible objects and describe them. A smartphone version of the app performs the same function using a mobile device’s camera: and even provides audible cues to let him know if he needs to move the camera in order to keep the item he’s photographing in the frame.

Here are a few use cases shown off in a demo video:

  • Take a picture of a restaurant menu with a smartphone and have the map read the headings or individual items out loud.
  • Tap the side of the smart glasses to identify the people in the room and give brief descriptions (like how old are the people, what’s their gender, and what emotions are they showing?)
  • Describe other people or objects that are within view of the camera, read other text.

Overall, this demo was probably one of the most compelling things I saw during Microsoft’s developer conference keynote, and it was the project which Microsoft used to conclude the event. But the company didn’t actually mention if or when the software will be available to the public.

CEO Satya Nadella did says that Shaikh is at the conference to network with other developers though, so maybe we’ll see this technology used in a variety of products from Microsoft and third-party developers in the future?

Support Liliputing

Liliputing's primary sources of revenue are advertising and affiliate links (if you click the "Shop" button at the top of the page and buy something on Amazon, for example, we'll get a small commission).

But there are several ways you can support the site directly even if you're using an ad blocker* and hate online shopping.

Contribute to our Patreon campaign


Contribute via PayPal

* If you are using an ad blocker like uBlock Origin and seeing a pop-up message at the bottom of the screen, we have a guide that may help you disable it.

Subscribe to Liliputing via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 9,498 other subscribers