cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
tareenmj
Helper I
Helper I

I'm trying to perform OCR on an image with Power Automate

As the title describes, I'm trying to do OCR on an image which a user submits from Microsoft Forms. The flow is as follows: (i) User submits a form (which includes questions and prompts the user to upload an image), (ii) Gather responses and perform OCR on the image, and (iii) Send an email with the responses and the parsed image. To accomplish this, I setup the following flow: 
The Flow ProcessThe Flow Process

 

With this, I am receiving the following error: 

Error messageError message

 

I'm not sure what the error is since I double checked and I did submit a png image. Any ideas on what I can do differently? 

1 ACCEPTED SOLUTION

Accepted Solutions
JoeF-MSFT
Power Apps
Power Apps

Hi @tareenmj - thanks for the question and cool use case! I didn't know files can now be submitted in Microsoft Forms! 💚

 

The add a screenshot... response contains a number of metadata on the file. There are two steps you need to add to your flow to get the image content.

 

1. A Parse JSON action. Select the add a screenshot... field for Content. And add the following for Schema

 

{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "name": {
                "type": "string"
            },
            "link": {
                "type": "string"
            },
            "id": {
                "type": "string"
            },
            "type": {},
            "size": {
                "type": "integer"
            },
            "referenceId": {
                "type": "string"
            },
            "driveId": {
                "type": "string"
            },
            "status": {
                "type": "integer"
            },
            "uploadSessionUrl": {}
        },
        "required": [
            "name",
            "link",
            "id",
            "type",
            "size",
            "referenceId",
            "driveId",
            "status",
            "uploadSessionUrl"
        ]
    }
}

 

2. Add a Get file content action from the OneDrive for Business connector to get the actual image that was uploaded on the Form. Use the id output coming from the 'Parse JSON' action for the File input.

 

3. You can then use the file content output from the previous OneDrive action in the AI Builder OCR action. 

 

JoeFMSFT_0-1651938228446.png

 

Let us know if this works for you!

 

View solution in original post

6 REPLIES 6
JoeF-MSFT
Power Apps
Power Apps

Hi @tareenmj - thanks for the question and cool use case! I didn't know files can now be submitted in Microsoft Forms! 💚

 

The add a screenshot... response contains a number of metadata on the file. There are two steps you need to add to your flow to get the image content.

 

1. A Parse JSON action. Select the add a screenshot... field for Content. And add the following for Schema

 

{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "name": {
                "type": "string"
            },
            "link": {
                "type": "string"
            },
            "id": {
                "type": "string"
            },
            "type": {},
            "size": {
                "type": "integer"
            },
            "referenceId": {
                "type": "string"
            },
            "driveId": {
                "type": "string"
            },
            "status": {
                "type": "integer"
            },
            "uploadSessionUrl": {}
        },
        "required": [
            "name",
            "link",
            "id",
            "type",
            "size",
            "referenceId",
            "driveId",
            "status",
            "uploadSessionUrl"
        ]
    }
}

 

2. Add a Get file content action from the OneDrive for Business connector to get the actual image that was uploaded on the Form. Use the id output coming from the 'Parse JSON' action for the File input.

 

3. You can then use the file content output from the previous OneDrive action in the AI Builder OCR action. 

 

JoeFMSFT_0-1651938228446.png

 

Let us know if this works for you!

 

Such a thoughtful and well-written answer. I didn't realize I needed to perform actions upon the uploaded file but thank you so much for the help! It worked out perfectly. Thanks!

JoeF-MSFT
Power Apps
Power Apps

Glad to hear you got it working - congrats @tareenmj! 🙂

 

I'm curious to learn more about your scenario (and if you can share) - what type of images are your users submitting through the Microsoft Form you are doing OCR? Is it a photo of a document or similar? 

Thanks! Absolutely, the problem I am trying to solve is actually a bit more complicated than just doing OCR. I'm trying to read OCR of screenshots: 

 

Service-Mode-With-Samsung-device.png

 

The problem I'm trying to solve is a bit more complicated than just OCR. I need to store the values in the screenshot as variables. So for example, the TAC of 3107 into a separate variable. Also, since the screenshots can vary quite a bit, I can't just build an AI function to extract certain pieces of the screenshot. Some screenshots will have more information than others. 

JoeF-MSFT
Power Apps
Power Apps

This is really interesting. Thanks for sharing @tareenmj!

 

Do the screenshots vary jut in the number of rows? If so, you could look at building an AI Builder Form Processing model and extract the table: https://docs.microsoft.com/en-us/ai-builder/create-form-processing-model#tag-tables

 

You could also look at combining OCR with AI Builder Entity Extraction. You can train an Entity Extraction model to extract certain elements from a body of text: Create an entity extraction custom AI model - AI Builder | Microsoft Docs

 

Not a problem, I'm glad you found it interesting. I was thinking of using an AI Builder Entity Extraction but thing is since the screenshots can vary, I'm not sure if this is the right step. The screenshots can vary in not only number of rows but also the title (for example screenshot A might contain value1 = 35 while screenshot B will contain value1-trio = 26C). I was thinking that I could parse the string obtained from OCR. In the case of the previously mentioned example, find the substring position at "value1=" and then adding a certain number to be able to get the value 35. Similarly, I would do another substring search for "value1-trio =" and then offset to get the value of "26C". Do you think this would be a better approach or could I get this done using the custom extraction? 

Helpful resources

Announcements
 WHAT’S NEXT AT MICROSOFT IGNITE 2022

WHAT’S NEXT AT MICROSOFT IGNITE 2022

Explore the latest innovations, learn from product experts and partners, level up your skillset, and create connections from around the world.

Register for a Free Workshop.png

Register for a Free Workshop

Learn to digitize and optimize business processes and connect all your applications to share data in real time.

Top Kudoed Authors
Users online (1,528)