cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
bolin-stratus
Continued Contributor
Continued Contributor

Targeting an UI Element by LegacyAccessible.Name?

I am using Inspect Object to view UI info for non-web windows and a button in the source window has been difficult to create a reliable selector.  I can use a UI Automation action and get it working, that is until I reboot and then the PAD flow won't work (dynamic values in my selector).  Can I target a button element using one of these other parameters like LegacyAccessible.Name?  If so, what would the selector look like?  
Power Automate Desktop - LegacyAccessibleName.PNG

1 ACCEPTED SOLUTION

Accepted Solutions
tkuehara
Solution Specialist
Solution Specialist

Hi @bolin-stratus,

 

I did the selector below and it worked, even after three restarts:

tkuehara_0-1624990905086.png


@bolin-stratus wrote:

Here is one of the selectors attempted.  I do understand that targeting an element by ID is always the best method.  


 Unfortunately, it's not always the case. If you inspect Pulse elements, you'll notice all buttons have the same ID:

tkuehara_1-1624991546561.png

tkuehara_2-1624992368501.png

 

View solution in original post

11 REPLIES 11
aminab-msft
Microsoft
Microsoft

hello @bolin-stratus 

 

thanks for writing here! can you tell me more about this statement: "I can use a UI Automation action and get it working, that is until I reboot and then the PAD flow won't work (dynamic values in my selector)."

 

I am unclear what is working and what is not. If the selector for this button has dynamic parameters that are well configured then the flow should catch the button all the time. If it catches sometimes but sometimes not, then maybe we need to revisit the selector. 

 

Could you please do the following for me: 

 

1. paste the selector as you currently built it: go to UI Elements > hit vertical ... menu for the UI element of the button > Edit Selectors > hit vertical ... menu near the selector > Edit Selector > copy and paste the string at the bottom of the screen. Like in the screenshot below.

 

aminabmsft_0-1624541115595.png

 

2. capture multiple times selectors for the button, by repeating this sequence multiple times: go to UI Elements > hit vertical ... menu for the UI element of the button > Edit Selectors > hit the descending arrow near the New button > click on Selector with Recapture > Ctrl-Click on the button

 

aminabmsft_1-1624541228033.png

 

 3. repeat step 1 with the newly captured selectors. 

 

This way we can compare the captured selectors, assess if there are changes from capture to capture and understand how to best build the selector for this button. 

 

You're almost there! 

 

Happy Automation,

Amina

@aminab-msft - Thank you for your response and detailed instructions.
I am using this Inspect tool to get all info on the appropriate Connect button.

PAD - Pulse Press Button Inspect UI Element.PNG

Here are all the steps with steps 6-11 the elements that have selector issues post reboot.
PAD - Pulse All actions.PNG
Here is where it is failing (as well all subsequent steps).

PAD - Pulse Press Button 1.PNG

Here is one of the selectors attempted.  I do understand that targeting an element by ID is always the best method.

PAD - Pulse Press Button selector.PNG

Here is the associated selector builder (attempt 1)
PAD - Pulse Press Button selector builder.PNG

Here is the associated selector builder (attempt 2)
PAD - Pulse Press Button selector builder 2.PNG
I performed the recapture starting with this Connect button followed by the remaining steps and it worked.  However, it did not work after a reboot (errors at this Connect button step 6).  I then performed recaptures again from step 6 on, and it produces the same select as before reboot. That is why I was starting to think creating selectors from the Inspect UI results might work, but I didn't know what a selector would look like for a UI window.
PAD - Pulse Press Button 2 recaptures.PNG

I compared the selector for the Connect button in step 6 using recapture (I recaptured it before and after reboot) and the selectors were identical.  Any recommendations are greatly appreciated.

bolin-stratus
Continued Contributor
Continued Contributor

@aminab-msft - Let me know if you require additional information. Thank you!

bolin-stratus
Continued Contributor
Continued Contributor

@aminab-msft  - So I upgraded to the Accessibility Insights for Windows tool and ran a test on the Pulse Connect button.  There were no errors but I don't know what to use for the selector.  Any insight is appreciated.
Pulse Button Accessibility Insights Report.PNG

Here is the Live Inspect.
Pulse Button Accessibility Insights Live Inspect.PNG

aminab-msft
Microsoft
Microsoft

hello @bolin-stratus 

 

We do not forget you. I have asked the Subject Matter Expert to help you out with selectors. They will be right there with you. Please ping this thread if you don't hear back from us by end of week. 

 

Happy Automation,

Your Power Automate team

@aminab-msft - Thank you for your response and I appreciate and you following up with the SME. Getting the answers to my selector questions will allow me to automate a lot more desktop UI functions.

tkuehara
Solution Specialist
Solution Specialist

Hi @bolin-stratus,

 

I did the selector below and it worked, even after three restarts:

tkuehara_0-1624990905086.png


@bolin-stratus wrote:

Here is one of the selectors attempted.  I do understand that targeting an element by ID is always the best method.  


 Unfortunately, it's not always the case. If you inspect Pulse elements, you'll notice all buttons have the same ID:

tkuehara_1-1624991546561.png

tkuehara_2-1624992368501.png

 

View solution in original post

NikosMoutzou
Microsoft
Microsoft

Hello @bolin-stratus .

 

From the experience you are describing, I infer that the selector for the specific UI element is dynamic and it is changing with every relaunch of the application.

 

In order to handle this, you have to create a new more robust UI selector, that will be able to locate the element even if the underlying structure of the element is dynamic. Any parts of the selector that are dynamic, should be removed. To achieve that, capture again the element after the failure, and compare the new selector with the old one to identify the differences like @aminab-msft indicated (copy/paste the new one and the old one into a notepad and then compare them). Please try to perform the comparison with the default selectors, as have been captured by Power Automate Desktop, before any editing.

 

After identifying the dynamic part of the selector, edit it to make sure that it contains only static elements/attributes that are not going to change. Also, you might have to use the operators into the Selector builder, as 'Starts with', 'Contains', etc. or regular expressions in order to specify the value of a selector and support dynamic values.

For example, the below captured UI element selectors indicate that the UI element has dynamic selector:

  • > custom[Class="ObjectEditor"] > custom[Id="AccountEditor"] > tab[Class="TabControl"] > tabitem[Name="Account Detail"] > text[Id="tbAccountId192"]
  • > custom[Class="ObjectEditor"] > custom[Id="AccountEditor"] > tab[Class="TabControl"] > tabitem[Name="Account Detail"] > text[Id="tbAccountId182"]

As you may notice, we have to support dynamic values for the leaf level attribute. Therefore, we have to change the 'Equal to' operator to 'Starts with' and delete the postfix number that is different in the two selectors:

NikosMoutzou_0-1625209503494.png

 

And the new selector:

  • > custom[Class="ObjectEditor"] > custom[Id="AccountEditor"] > tab[Class="TabControl"] > tabitem[Name="Account Detail"] > text[Id^="tbAccountId"]

 

Also, in some cases you might have to use variables in the selectors. The values of these variables should be set into the desktop flow before the selector is used.

 

Example:

  • Set variable %ID_var% in desktop flow.
  • Use it in the selector as:
    • > custom[Class="ObjectEditor"] > custom[Id="AccountEditor"] > tab[Class="TabControl"] > tabitem[Name="Account Detail"] > text[Id="tbAccountId%ID_var%"]

 

Moreover, although using the ID attribute is usually the best way to proceed, you have to ensure that the selector identifies one and only one element in the window. If all 'Connect' buttons in the window have the same value for the ID attribute, then you have to insert additional attributes in order to make the selector unique.

 

I hope the above helps.

bolin-stratus
Continued Contributor
Continued Contributor

@tkuehara - Thank you for the great info.  Unchecking the #1 selector (ie at the top), fixed the issue for all the selectors.  I have verified that after reboot, these selectors still work.  Thanks again.
PAD UI selector solution for Pulse.PNG

@NikosMoutzou - This information is very useful and I will refer to this frequently moving forward!

aminab-msft
Microsoft
Microsoft

So great to read this. Feel free to mark the answer as the solution.

 

happy automation!

your Power Automate team

Helpful resources

Announcements
MPA_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

V3_PVA CAmpaign Carousel.png

Community Challenge - Giveaways!

Participate in the Power Virtual Agents Community Challenge

Carousel 2021 Release Wave 2 Plan 768x460.jpg

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

PowerPlatform 768x460.png

Microsoft Learn

Check out our new Discover Your Career Path blog post series and get all the details.

Top Solution Authors
Top Kudoed Authors
Users online (3,008)