form recognizer - multiple label custom model - parsing the JSON

[nightfall]

Hello everybody,

i have trained a custom model for recognizing a tax payment form. Using the HTTP POST and GET actions i manage to get the results. 

I am however struggling in parsing the JSON. As  mentioned in the title, the model has quite an amount of labels (the tax form has 4 sections, each section has at least 5 rows and 4 columns).

As I am unfortunately a tax advisor and not a coder, i am struggling with doubts on how to parse the JSON:

1. My first doubt starts already with the "generate from sample" option in the "parse JSON" step. While RTFM-ing for clues every guide basically suggests to run the flow and then copy-paste (with some correction) the output. However, not all the labelled fields will be used, this means that some object in the array won't have a property at all.  SHould i add them manually?
2. Every labelled field's property in the array has the same name: how can i know which valueString is related to e.g. L1DEB and which to TOT?

Another doubt that I have is not directly connected to flow per se, and has to do with how many pages can i send to the model in a certain amount of time. As an example, in a current flow i send around 160 single paged PDF all together. Given that i am currently in the F0 plan, is there a limit of files that i can send at the same time? if yes, how can i prevent that all pages will be sent to the form recognizer, if the flow trigger is "when a file is created in a folder in sharepoint"??

Sorry for the long questions...!!!

[braydenlarson]

Hello Nightfall,

When you say a trained custom model, are you speaking about a AI Builder model that is designed to pull the fields for you?

The largest benefit of the AI Builder inside of Power Automate is you can set it up to automatically generate the dynamic content you want to grab. you can select and label the fields you want it to grab and as long as the model is given enough test data it should be usable under the AI Builder connector in your flow.

Example:

The big thing is that AI Builder does not have a trigger so you would have to set it up in a way where when a file is received it can trigger a flow that will grab said file and run it through your selected model.
You can even setup multiple models based off the forms you use and when the flow is triggers it looks at values such as the file name or form name to identify what model it runs off of.

Otherwise, if you still need to use parse JSON, the suggestion of running a sample is needed to generate the different groups it uses. After that if it doesn't distinguish the dynamic content fields(I.E. labels them unique enough to identify) you can setup variables that you input them into and then later on in your flow to help better distinguish which field is what.

Finally as for capacity,

you will want to review the documentation and AI Builder calculator linked below to help determine what you need

AI Builder Licensing - AI Builder | Microsoft Docs

AI Builder Calculator | Microsoft Power Apps

have a good day,

[sakula1996]

Hi @nightfall ,

If you're using SharePoint list and adding attachments to it. You can use condition to a SharePoint trigger. Check the image below:

[sakula1996]

Hi @nightfall ,

In the dynamic content you should see two different variables. Just search for 

TOT value & LTOT value

You can use those values.

[nightfall]

Hi @braydenlarson,

thank you for your reply, and sorry for my late answer.

I mention a custom model because i am using the custom model service of the form recognizer module of Azure's cognitive services. There used to be a Power Automate ad-hoc connector, but it's been deprecated, therefore in order to connect to it one must use the HTTP connector.

I have tried AI Builder, however i was not really satisfied by it, notwithstanding the fact that the engine is the same, even though with the introduction of the "collections".

Best regards,

Vittorio 

[nightfall]

hi @sakula1996 , thank you for your hints. Sorry for the late reply, but i was spending the last days experimenting. Indeed, as you correctly point out, if i use directly the values such as TOT or LTOT etc, flow puts an apply for each step, that will return the whole content of the object:

in order to get e.g. the corresponding amount of LTOT (928), i need to use the valueString variable. this would be ok for me, if i had only these 2 fields.

However, as i have mentioned, i have many more fields (28), and usually only 5-6 of these fields are used, therefore it would be very complicated and inefficient to put them manually.

In my mind, the first parse json should parse the documentResults array, and then i'd make another apply to each of the various fields, checking if the array is null and, if not, would pass the name of the field and the amount to e.g. a sharepoint list.

however, the first apply to each finds only one item, whereas the second parse json (that would parse the output of documentResults returns the following error:

The execution of template action 'Apply_to_each_2' failed: the result of the evaluation of 'foreach' expression '@body('ParseJson')' is of type 'Object'. The result must be a valid array.

Because indeed as shown in the picture above thetype is object and not array.. 

The solution is for sure straightforward, but maybe i'm trying to bite more that i can chew..

thank you in advance,

vittorio