Not able to detect buttons inside the pane

[Anonymous]

Hi everyone,
I just started using Power automate last week and im trying to click a button inside a UI application but the AI doesnt seem to recognize anything inside that big pane window. Is power automate limited or is there a workaround this. please and thanks

[fraenK]

There is no AI behind that function 😉

You can check with Inspect.exe wich elements you can access directly:

https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects

Otherwise you'll need to use keyboard/send keys, text (OCR) or image based actions:

https://docs.microsoft.com/en-us/power-automate/desktop-flows/actions-reference/mouseandkeyboard

[Anonymous]

HI fraenk,
Thanks for your time. Instead of Inspect.exe I installed accessibility insight but im not sure how I can use that in combination with power automate to create a flow. Can you please guide me to any links that has similar topics related. Please and Thanks

[fraenK]

https://docs.microsoft.com/en-us/previous-versions/troubleshoot/winautomation/process-development-ti...

[kmur]

I'm still seeing only the pane on accessibility insight and not the buttons. Which is frustrating because literally yesterday I could access the buttons for UI automation, and now for some reason with no updates or even a restart I can only access the panes.

Any idea why this could be happening?

[kelway]

I'm having this same problem now and I wonder if you found a reason?

My flow had reference to a specific field as my screenshot shows, and today when I ran it, the step wasn't able to find that Gui Password field... and you can see below it that when I try to select the field again, I only can get a reference to the greater Pane. Very frustrating... I just reinstalled PAD too. In my case it's with SAP client.

Thanks for any information.

[johnlouie08]

I was wondering if this problem has been solved because right now, I am encountering the same issue on automating SAP. I've captured some objects on SAP while developing an RPA but now that it is done, the captured objects are showing an error because the huge Pane is the only thing that is being recognize in SAP UI.

[kelway]

I don't recall exactly, but it definitely just worked again at some later point. As I consider it now after another problem with SAP I'll explain in a bit, I wonder if it's related to SAP Server Scripting being enabled for the user or the SAP GUI client. I think I may have been working with an RPA account that already had server side scripting and I may have been using my own account at the point it failed.

WIth SAP, I've found clear behavioral differences when clicking and typing manually vs letting the bot do it with my personal SAP account. I can press Shift F9 to export a report and the dialogue is completely different if the bot does it or if I do the whole process manually using the same SAP account. If, however, I use an account created specifically for RPA job (and assume they set SAP Server Scripting for it), then it works manually OR with the same RPA job. This one caused me some extreme confusion and frustration!

Anyway, good luck. Maybe reboot a few times 😉

[Bretzeli]

Hello,

Don't use it simply BETA!

I been playing around with Power Automate Desktop for 2 Weeks now and it's just awfull. Lost Data because of Crash W10 Developer client an not able to save local (Just code COPY + PASTE), Not able to find UI from their own software like MS FS 2020. OCR works next minute not with same resolution, Text OCR works next day not with same resolution. Not able to set LANGUAGE (Just by language Dispplay APP OS) etc. etc.

The solution is horrible laggy and we will never use in Enterprise customer. We have a DSL Business line here and on 8 CORE CPU W10 22H2 extreme lag.

Maybe we can at least REBOOT/Restart Windows OS with the Microsoft Automation? Or maybe more reliable with just a shell command like:

shutdown /s /t 600 /c "Everything works reliable except Power Automate"

Not Powershell, Not M365, Not Power Automate? Just plainw roking and boring old style?

Or maybe use AUTOIT.exe which folks use since 20 years and works fine?

Greerings from Switzerland