cancel
Showing results for 
Search instead for 
Did you mean: 

Build your very own LEGO Inventory app using PowerApps, Flow and Microsoft Cognitive Vision Services

Recently, I’ve been intrigued by Microsoft Cognitive Vision services. This Machine-Learning functionality can extract rich information

from images to categorize and process visual data – and machine-assisted moderation of images to help curate your services. Now, as

some of you may already know, I’m a huge LEGO fan and I come from a family whose been playing with LEGOs for three generations.

One of the struggles each LEGO fan faces is identifying a part number. I took on this struggle as a challenge and have come up with a

Proof Of Concept using Microsoft PowerApps and Flow’s integration with Microsoft Cognitive Vision Services’ API. I introduce to you

my LEGO Inventory App.

 

Before I started building the app I had to find an Application Programming Interface (API) which would let me access a LEGO inventory

to gather information I need. Fortunately one of my personal favorite websites, Rebrickable.com already provided this API.

Rebrickable.com provides a number of API/Web Services to assist developers build their own websites or apps which use the

Rebrickable database. To be able to use these, you must first Generate an API Key which is free but does require registration

 

 

Rebrickable.png

 

My first iteration of this app involved me using Cognitive Vision's Optical character recognition (OCR) which would detect the LEGO

part numbers extract the recognized numbers into a machine-readable character stream. This gave me the flexibility of taking photos

of text instead of copying to save time and effort. However, I did run into a road-block. You see the part numbers in the LEGO blocks

are so tiny that it was challenging to take a zoomed-in picture. Hence I added an option to type in a part number as well but I did create

some modified blocks to test the OCR feature.

 

This video demonstrates how the first iteration of this app works

 

LEGO Inventory App using PowerApps and Flow

 

 

The first iteration works well, however, there were already two apps available which could do the exact same. Rebrickable Shopper

by Rebrickable is a mobile app where you simply scan the bar code on a LEGO® box and see a wealth of information to help you decide

if you should buy it. The Brick Data by Jean-Michel Agresti is an iPhone app perfect to browse LEGO Sets on your mobile device, old and

new. With these two existing apps already available any solution that I build would have to that major WOW factor, something that

makes it easier. This is when Cognitive Custom Vision Services had my complete and undivided attention. The simple concept of upload,

train and evaluate is brilliant. In addition, through it's active learning images evaluated through your custom vision model become part

of a feedback loop you can use to keep improving your classifier. Hence I decided to pursue that path and came up with my LEGO

Inventory App 2.0

 

This video demonstrates how the 2.0 iteration of the app works

 

Microsoft PowerApps with Cognitive Custom Vision

 

 

From a design and build standpoint the difference between OCR and Custom Vision was the change in the FLOW.

The below screenshot displays how the OCR and Custom Vision action is called and how it's values are us

 

Capture2.PNG

 

Currently Custom Vision is in preview hence there are some limitations but the overall P.O.C. was very successful.

 

URL:

https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/

https://azure.microsoft.com/en-us/services/cognitive-services/custom-vision-service/

Meet Our Blog Authors
  • Co-founder of https://plumsail.com, Office 365 and SharePoint expert. Passionate about design and development of easy to use, convenient and flexible products.
  • I'm the owner of ThriveFast, an Office 365 consulting company. I'm a developer and have been working with SharePoint since 2007.
  • 7x Microsoft Business Solutions MVP (CRM)
  • Solution Architect with Slalom, and organizer of the Boston Office 365 User Group, and long term SharePoint/Office 365 veteren. Find more at http://www.davidlozzi.com. Follow @DavidLozzi
  • I'm keen in MS technologies, SharePoint, Office 365 and development for them
  • Michelle is an Office 365 solution architect in Twin Cities, MN. She has been delivering business collaboration solutions for years with her focus on SharePoint and Office 365. Michelle is a recent board member of the Minnesota Office 365 User Group and has been a member of the SharePoint community since 2009. She is a frequent speaker at MNSPUG and SharePoint Saturday and co-chaired the Legal SharePoint User Group for 4 years. Her most frequent projects have involved rolling out a large deployment of Office 365, SharePoint Online intranet, build of a "CHAMPS" Office 365 user adoption program and most recently, SharePoint On-Premise to Online Migration. Michelle is very excited about cloud technology as it is shifting her IT Pro focus to collaboration strategy and technical adoption.
  • I'm a Microsoft Office Servers and Services MVP with a special interest in SharePoint, Office 365, Microsoft Flow, Microsoft Teams and PowerApps. I work at Triad Group Plc ( http://triad.co.uk)
>