powerapps computer vision connector

[RusselThomas]

Does anyone have a working example of using the Computer Vision API connector in PowerApps with the camera image control? I'm aware it's in preview, but presumably there's a reasonable use case for preview...

I have used Flow to achieve what I'm trying to achieve, I'm looking for an example that does not involve Flow or a custom connector.  If I have to use Flow or a custom connector then it seems the PowerApps connector is a bit redundant? at least a bit limited as far as the Camera control is concerned...

Kind regards,

RT

[v-xida-msft]

Hi @RusselThomas,

Could you please share a bit more about your scenario?

Do you want to analyze the image captured via Camera control using the Computer Vision API connector within your app?

I have made a test on my side, please take a try with the following workaround:

Add a Camera control, Button control and a Gallery control within the screen. In addition, add the Computer Vision API connector as a data source within your app.

Set the OnSelect property of the Button control to following formula:

ClearCollect(ImageCollection,ComputerVisionAPI.AnalyzeImage("Image Content",{language:"en",visualFeatures:Camera1.Photo}))

Note: The Camera1 represents the Camera control within my screen. Using a collection to store the result the ComputerVisionAPI.AnalyzeImage function returns.

Set the Items property of the Gallery control to following formula:

ImageCollection

More details about the Computer Vision API connector, please check the following article:

Computer Vision API connector

Best regards,

Kris

[RusselThomas]

Thanks Kris, this is precisely the use case 🙂

I was battling to find out where to put the image content, and in what format - what I read about visualfeatures didn't really make me think it should go there...

Carlosag's reference

Parameters:

Name	Type	Summary	Required	Related Action
language	string (Language)Values: [en, zh]	The service will return recognition results in specified language.	False
visualFeatures	string (Visual Feature Options)	A string indicating what visual feature types to return. Multiple values should be comma-separated.	False
format	string (Image Source)Values: [Image Content, Image URL]	Source of the image - either included or by reference url.	True
Image	AnalyzeImageParameterImage		True

and it doesn't even register on the MS Docs reference, although it's obviously in the object hierarchy:

I tried using your fomula, but I still get imageformat errors;

ComputerVisionAPI.AnaIyzeImage failed: {

"code": "InvalidlmageFormat",

"requestld": "70526a86-b4e5-4bbc-8ea4-0748c775cb2a",

"message": "Input data IS not a valid image."

Is yours working?

Kind regards,

RT

[v-xida-msft]

Hi @RusselThomas,

Just an update -- Please take a try with the following formula on your side:

ClearCollect(ImageCollection,ComputerVisionAPI.AnalyzeImage(Camera1.Photo,{language:"en",visualFeatures:"Tags"}))

Please check and see if the following article would help in your scenario:

https://westus.dev.cognitive.microsoft.com/docs/services/56f91f2d778daf23d8ec6739/operations/56f91f2...

Best regards,

Kris

[RusselThomas]

Busy going through the documentation, same error on the updated formula though.

From what I read somewhere yesterday, the vision service expects binary input - from what I can see, the camera puts out base64 encoded jpg....?

[Edit]

Also referenced in the link you sent;

• For an image URL: Content-Type should be application/json
• For a binary image data: Content-Type should be application/octet-stream or multipart/form-data

[v-xida-msft]

Hi @RusselThomas,

Yes, your are right. The image the the Camera control captured is a base64 Data URI, the format of the base64 Data URI as below:

data:[<media type>][;base64],<data>

Please check the following blog for more details:

https://medium.com/dtlpub/adding-camera-capability-to-the-powerapps-visitor-registration-app-e5f8402...

If you want to convert a base64 Data URL into an binary data, I afriad that there is no way to achieve your needs in PowerApps currently.

If you would like this feature to be added in PowerApps, please submit an idea to PowerApps Ideas Forum:

https://powerusers.microsoft.com/t5/PowerApps-Ideas/idb-p/PowerAppsIdeas

As an alternative solution, you could consider take a try to pass the image (base64 Data URI) captured via the Camera control within your app to a Microsoft Flow, then within your flow, convert the base64 Data URI into an Binary data.

Please check and see if the following blog would help in your scenario:

https://www.techmikael.com/2017/05/an-even-more-clever-workaround-for.html

Best regards,

Kris

[RusselThomas]

Thanks Kris,

I use the Flow uritobinary expression often as it works well for saving images to SharePoint, but returning the binary to PowerApps and submitting it to the Computer Vision Connector hangs my studio session (using the "Respond to PowerApps" flow action).

Besides this, as I said initially - if I'm going to Flow once, then I may as well process everything (including the CVAPI call) from there and just return the results of the call to PowerApps (which I'm doing now anyway).  This then obviates the need for having the PowerApps CVAPI connector in my app - but then surely I'm missing a trick here somewhere otherwise why do we have the "Image Content" functionality in the connector?

I've also tried sending content from the image upload control and I get the same results, so at this point I'm pretty much back to my original question - can I use the PA Computer Vision connector for sending any image content from PA to the connector, or is it only useful for URL's of images stored somewhere else?  (Something I haven't tested yet, but I'm assuming that at least works)

Kind regards,

RT

[ashay-maheshwar]

@v-xida-msft, I coped your example, but I can see an error , "Invalid Image format", input data is not a valid image". Can you share your insights on this, Thanks

[MJDuane]

Hi, has anyone actually have a working example of computer vision where you take a picture then have it analyzed and the results presented in a gallery it does not work with the API and when I use flow I get an invalid argument error.

[ashay-maheshwar]

Hello @MJDuane , 

You can try this - 

a. Use Camera control to take a picture

b. Use Azure Blob Connector to upload Camera Image to Blob

c. Create a Computer Vision Cognitive Service on Azure and write your logic using available SDK (I used Python SDK and created a project on Custom Vision for object detection, for example laptop, mouse, etc)

d. Expose your code in point c. as a RESTFul Service

e. Create a custom connector in PowerApps

f. Get the desired response

I hope this helps. In case you need more details, let me know.