How to Manage Handwritten Feedback with Microsoft Flow
Nowadays everything is digital. Almost. Still, many times when I am asked to give a feedback, I get a pen and paper. And I have only one thing on my mind.
Is there someone who reads all of these handwritten feedbacks?
And even worse, does she/he type all feedbacks into some feedback management system?
Hope not. But I feel so sorry that I build simple Flow for managing handwritten feedbacks.
Handling course feedbacks
Let's think about trainers giving different courses. After session participants fulfill feedback forms (in a paper). But after that, the trainer takes a photo from each of the feedback forms and sends them to handle.feedbacks@mytenant.onmicrosoft.com.
There is a Flow waiting for those emails. With every email, it reads texts in images (using Azure Computer Vision) and makes sentiment analysis for them. Results are stored in a SharePoint document library.
Sounds easy. But is it?
SharePoint document library for storing feedbacks
First, we need a place where to store all the feedbacks.
SharePoint document library is the perfect solution for this. Create a new one and add the following columns to it.
- Course (Name of the course feedback is related to)
- Content (Handwritten text recognized from paper form)
- Sentiment (Tone of the feedback)
- Trainer (Who was the trainer on course feedback is related to)
Azure services
Our solution utilizes Azure services. Let's create them. If you don't have an Azure account, you can always use trial account.
Click Create a resource, search for the Computer Vision API and create a one.
After that create a Text Analytics API.
Then pick a key from both of the created services. You will need them later.
Flow
Our Flow starts every time handle.feedbacks@mytenant.onmicrosoft.com account gets an email.
In the beginning, we initialize variable for parsed text.
Then we get all email attachments.
Flow doesn't have a connector for the Computer Vision API. Therefore we make an HTTP call to a Computer Vision API service we just created. Use the service key you picked up earlier as a value of Ocp-Apim-Subscription-Key.
You get a return message from Computer Vision API in JSON format. It looks like this.
If you want to understand in more detail how to use handwritten text recognition service, read API description. Shortly, calling the service doesn't return recognized text but the URL (Operation-Location). You can use the URL for asking actual results. The URL is valid for 24 hours.
So, we need to parse out that Operation-Location value. Fortunately, Flow has the parse JSON action. You can use sample payload and Parse JSON action generates a schema for you. Simple.
After that, we can easily use output values of parse JSON action in our Flow.
Since image analysis may take a while, we add 2 minutes delay before trying to get the parsed text.
After waiting, the results are retrieved with another HTTP request. Use the Get operation with Operation Location URL. Ocp-Admin-Subctiption-Key is the same as in the previous action.
And return value is parsed by using parse JSON action.
After parsing JSON the written text does not exist in a single variable (which would be way too easy). Words are divided into lines.
That’s not a problem. Just add words to the variable line by line.
Now we have handwritten text in our variable.
The last thing we do is a sentiment analysis for the given feedback. This can be done with the Sentiment Analyse action.
Almost ready! Now, we know the content and sentiment of the handwritten feedback. This information should be saved in our document library.
First, we create a file.
After that, we update file properties (= our document library columns).
At the end remember to initialize handwritten text variable for next attachment.
When we are ready our flow looks like this.
Does it work?
Surprisingly well. Let's make a few feedbacks and see what happens.
Take a photo of each feedback form. After that, send them to the email address you used in "When a new email arrives" -action. You can use the course name as a title.
After waiting for a few minutes, items appear in the course feedback document library!
Comments
-
How to Manage Handwritten Feedback with Microsoft Flow
That's true!
If the requirement is (I don't know if it is) to handle existing images on the background (with Flow for example), then PowerApps cannot be used.
-
How to Manage Handwritten Feedback with Microsoft Flow
PowerApps has a Barcode/QR-code reader reader that is pretty good. you can read more about it here - https://powerusers.microsoft.com/t5/PowerApps-Community-Blog/The-current-state-of-working-with-barcode-and-QR-code-images/ba-p/133772
-
How to Manage Handwritten Feedback with Microsoft Flow
Hi!
Yes and no... It depends very much about images you have. There is OCR service on Azure computer vision service (https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/).
You can use it to recognize text from the image. But if there are several different texts in the image, then you have to find out by yourself which text is the right one. Which is a serial number and which is product name for example.
Also, there isn't (currently) service in Azure that recognizes QR code from the image.
-
How to Manage Handwritten Feedback with Microsoft Flow
Very impressive. I have a client that would like to OCR content of a digital photo taken from a mobile camera (like an equipment tag, or a serial number in box). Could this be used to read QR Code as well?
Thanks for writing this up in the community.
-
How to Manage Handwritten Feedback with Microsoft Flow
But this was an excellent point. You should always check out what you get from existing connectors. And if there wasn't connector/action you need a month ago, it doesn't mean that it isn't there today.
-
How to Manage Handwritten Feedback with Microsoft Flow
Oh ok! got it! Thanks for the clarification.
-
How to Manage Handwritten Feedback with Microsoft Flow
Yes, that's true! Unfortunately, there is no action for analyzing handwritten text.
-
How to Manage Handwritten Feedback with Microsoft Flow
Great post! Just a note : Flow does have Computer Vision API connector. That could help in getting some of the above steps done more quickly.
*This post is locked for comments