cancel
Showing results for 
Search instead for 
Did you mean: 

How to Manage Handwritten Feedback with Microsoft Flow

Nowadays everything is digital. Almost. Still, many times when I am asked to give a feedback, I get a pen and paper. And I have only one thing on my mind.

Is there someone who reads all of these handwritten feedbacks?

And even worse, does she/he type all feedbacks into some feedback management system?

Hope not. But I feel so sorry that I build simple Flow for managing handwritten feedbacks.

Handling course feedbacks

Let's think about trainers giving different courses. After session participants fulfill feedback forms (in a paper). But after that, the trainer takes a photo from each of the feedback forms and sends them to handle.feedbacks@mytenant.onmicrosoft.com.

There is a Flow waiting for those emails. With every email, it reads texts in images (using Azure Computer Vision) and makes sentiment analysis for them. Results are stored in a SharePoint document library.

Sounds easy. But is it?

SharePoint document library for storing feedbacks

First, we need a place where to store all the feedbacks.

SharePoint document library is the perfect solution for this. Create a new one and add the following columns to it.

  • Course (Name of the course feedback is related to)
  • Content (Handwritten text recognized from paper form)
  • Sentiment (Tone of the feedback)
  • Trainer (Who was the trainer on course feedback is related to)

nc3a4yttc3b6kuva-2018-10-3-kello-17-01-12.png

Azure services

Our solution utilizes Azure services.  Let's create them. If you don't have an Azure account, you can always use trial account.

Click Create a resourcesearch for the Computer Vision API and create a one.

nc3a4yttc3b6kuva-2018-3-17-kello-17-21-55.png

 

After that create a Text Analytics API.

nc3a4yttc3b6kuva-2018-3-17-kello-17-21-55.png

 

Then pick a key from both of the created services. You will need them later.

nc3a4yttc3b6kuva-2018-3-17-kello-17-21-55.png

 

Flow

Our Flow starts every time handle.feedbacks@mytenant.onmicrosoft.com account gets an email.

In the beginning, we initialize variable for parsed text.

flow1.png

 

Then we get all email attachments.

flow2.png

 

Flow doesn't have a connector for the Computer Vision API. Therefore we make an HTTP call to a Computer Vision API service we just created. Use the service key you picked up earlier as a value of Ocp-Apim-Subscription-Key.

flow3.png

 

You get a return message from Computer Vision API in JSON format. It looks like this.

nc3a4yttc3b6kuva-2018-3-18-kello-11-34-02.png

 

If you want to understand in more detail how to use handwritten text recognition service, read API description. Shortly, calling the service doesn't return recognized text but the URL (Operation-Location). You can use the URL for asking actual results. The URL is valid for 24 hours.

So, we need to parse out that Operation-Location value. Fortunately, Flow has the parse JSON action.  You can use sample payload and Parse JSON action generates a schema for you. Simple.

flow45png.png

After that, we can easily use output values of parse JSON action in our Flow.

Since image analysis may take a while, we add 2 minutes delay before trying to get the parsed text.

flow6.png

After waiting, the results are retrieved with another HTTP request. Use the Get operation with Operation Location URL.  Ocp-Admin-Subctiption-Key is the same as in the previous action.

 

And return value is parsed by using parse JSON action.
flow7.png

 

After parsing JSON the written text does not exist in a single variable (which would be way too easy). Words are divided into lines.

That’s not a problem. Just add words to the variable line by line.

flow8.png

Now we have handwritten text in our variable.

The last thing we do is a sentiment analysis for the given feedback.  This can be done with the Sentiment Analyse action.

flow9.png

 

Almost ready! Now, we know the content and sentiment of the handwritten feedback. This information should be saved in our document library.

First, we create a file.

flow.png

 

After that, we update file properties (= our document library columns).

flow10.png

 

At the end remember to initialize handwritten text variable for next attachment. 

flow11-2.png

 

When we are ready our flow looks like this.

flow-overview.png 

 

Does it work?

Surprisingly well. Let's make a few feedbacks and see what happens.

Take a photo of each feedback form. After that, send them to the email address you used in "When a new email arrives" -action. You can use the course name as a title.

send-fedbacks-with-phone2.jpg

 

After waiting for a few minutes, items appear in the course feedback document library!

document-library-with-parsed-texts.png

 

Comments

Great post! Just a note : Flow does have Computer Vision API connector. That could help in getting some of the above steps done more quickly.

2018-10-09 14_45_45-Create your flow _ Microsoft Flow.png

Yes, that's true! Unfortunately, there is no action for analyzing handwritten text. 

Oh ok! got it! Thanks for the clarification.

But this was an excellent point. You should always check out what you get from existing connectors. And if there wasn't connector/action you need a month ago, it doesn't mean that it isn't there today.

 Very impressive.  I have a client that would like to OCR content of a digital photo taken from a mobile camera  (like an equipment tag, or a serial number in box). Could this be used to read QR Code as well?

 

Thanks for writing this up in the community.

Hi!

 

Yes and no... It depends very much about images you have. There is OCR service on Azure computer vision service (https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/).

 

You can use it to recognize text from the image. But if there are several different texts in the image, then you have to find out by yourself which text is the right one. Which is a serial number and which is product name for example.

 

Also, there isn't (currently) service in Azure that recognizes QR code from the image.

PowerApps has a Barcode/QR-code reader reader that is pretty good. you can read more about it here - https://powerusers.microsoft.com/t5/PowerApps-Community-Blog/The-current-state-of-working-with-barco...

That's true!

 

If the requirement is (I don't know if it is) to handle existing images on the background (with Flow for example), then PowerApps cannot be used. 

 

 

 

Meet Our Blog Authors
  • Working daily with Microsoft Cloud to deliver the needs of my company, my customers and various Microsoft communities and forums. | Office 365 | Flow | PowerShell | PowerApps | SharePoint |
  • Co-founder of https://plumsail.com, Office 365 and SharePoint expert. Passionate about design and development of easy to use, convenient and flexible products.
  • Microsoft Business Apps MVP. Owner of ThriveFast, an Office 365 consulting company.
  • 7x Microsoft Business Solutions MVP (CRM)
  • I'm keen in MS technologies, SharePoint, Office 365 and development for them
  • Daniel is a Business Productivity Consultant & Microsoft Business Solutions MVP who is very enthusiastic about all things Office 365, Microsoft Flow, PowerApps, Azure & SharePoint (Online). Since the preview, Daniel has been working with Microsoft Flow and later on with Microsoft PowerApps. That led to him being awarded an MVP Award for Business Solutions. He loves to blog, present and evangelize about improving productivity in the modern workspace with these amazing tools!
  • Michelle is an Office 365 solution architect in Twin Cities, MN. She has been delivering business collaboration solutions for years with her focus on SharePoint and Office 365. Michelle is a recent board member of the Minnesota Office 365 User Group and has been a member of the SharePoint community since 2009. She is a frequent speaker at MNSPUG and SharePoint Saturday and co-chaired the Legal SharePoint User Group for 4 years. Her most frequent projects have involved rolling out a large deployment of Office 365, SharePoint Online intranet, build of a "CHAMPS" Office 365 user adoption program and most recently, SharePoint On-Premise to Online Migration. Michelle is very excited about cloud technology as it is shifting her IT Pro focus to collaboration strategy and technical adoption.
  • I'm a Microsoft Office Servers and Services MVP with a special interest in SharePoint, Office 365, Microsoft Flow, Microsoft Teams and PowerApps. I work at Triad Group Plc ( https://triad.co.uk)
  • Passionate #Programmer #SharePoint #SPFx #Office365 #MSFlow | C-sharpCorner MVP | SharePoint StackOverflow, Github, PnP contributor