cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Anonymous
Not applicable

no unique UI selector, is there a way to read the screen to select element?

hi, I am trying to automate a process in Agile PLM to download attachments. I can select the text field and do a search, but when it return, there are multiple fields in the table :

 

Capture1.JPG

 

I want to click on the dot with the row that status is active (2nd row in the picture), is there a way to do that? I tried to extract data from web but I dont see a table to loop....

 

And lets say if I can do this, the next step is to select a pdf file to download. When I add the UI elements the selectors for the jpg link and pdf link are identical, is there a way to read the text and click the link that contains pdf?

 

Capture2.JPG

1 ACCEPTED SOLUTION

Accepted Solutions
tkuehara
Super User
Super User

Hi @Anonymous 

 

Short answer: yes, you can.

 

For those kind of solutions, I'd go for JS. Idk how familiar you are with JS but here it goes:

tkuehara_1-1628779313185.png

The above is a sample code (changed the element identifiers and comments). You add it in this action:

tkuehara_2-1628779606489.png

Then you apply the same logic for the second part of your code: if you get "true" from this JavaScript then it found the desired row and clicked the "dot" icon; in other words, you can now proceed to identify the pdf row and click the respective download button (assuming you have only one PDF per table).

 

Long answer: you could technically do that with elements as well, but it is a hassle: basically, you'll need to map at least two elements (one for the label/text and another one for the links... for the first part, and then another pair for the second table) also you'll need to extract the number of rows either by retrieving them as a data table to get the number of rows or... yes, using JS to do the same. Finally, you edit the elements to make them dynamic, so when you iterate the rows it updates the element's reference. Example:

tkuehara_7-1628781127006.png

 

View solution in original post

7 REPLIES 7
MichaelAnnis
Super User
Super User

There is image based processing.  The warning I provide is that for image based processing to work, anyone using the flow should be using the same sized monitor and the same display ratio as when the bot was originally set, and should not be remoted in.  The slightest variances can throw off image identification.  With that said, it is very accurate, very fast, and works very well when it seems like there is no other option. 

Anonymous
Not applicable

I have been testing with the OCR method but it is pretty slow (takes a couple sec to find and move mouse to the text) and it doesnt always find it on the first try too

One of the biggest helps to OCR is to create the Tesseract Engine first, use a multiplier of 3 and 3.  This basically zooms in on all the letters increasing the accuracy greatly; however, it makes it slower, because now there are more pixels to go through.  I have seen other flows that are intuitive in this, but they take longer to build, so they will start with a multiplier of 1 and 1, if they don't find what they need, they have an error subflow that increases the multiplier to 2 and 2, and tries again.  

Daniel_Pa
Resolver II
Resolver II

You could just use Move mouse to Image, and set an offset to the left, so you hit the link. Simple and pretty fast. You might have to tweak the tolerance.

Be aware that the size of the window must be static then.

@Anonymous - here you have the below options

1. if this is a web application, go to HTML code and trace the path

2.use the send keys options ( same like key board and their short cuts )*

3. use the mouse click options along with X and Y co-ordinates**

4.use mouse and send keys option

5. OCR with mouse or key board options

 

* - works perfect in unattended mode and risk in attended mode

** - as mentioned above by @Daniel_Pa  , should be screen size static always. 

tkuehara
Super User
Super User

Hi @Anonymous 

 

Short answer: yes, you can.

 

For those kind of solutions, I'd go for JS. Idk how familiar you are with JS but here it goes:

tkuehara_1-1628779313185.png

The above is a sample code (changed the element identifiers and comments). You add it in this action:

tkuehara_2-1628779606489.png

Then you apply the same logic for the second part of your code: if you get "true" from this JavaScript then it found the desired row and clicked the "dot" icon; in other words, you can now proceed to identify the pdf row and click the respective download button (assuming you have only one PDF per table).

 

Long answer: you could technically do that with elements as well, but it is a hassle: basically, you'll need to map at least two elements (one for the label/text and another one for the links... for the first part, and then another pair for the second table) also you'll need to extract the number of rows either by retrieving them as a data table to get the number of rows or... yes, using JS to do the same. Finally, you edit the elements to make them dynamic, so when you iterate the rows it updates the element's reference. Example:

tkuehara_7-1628781127006.png

 

Anonymous
Not applicable

Hi @tkuehara 

 

Thanks for the JavaScript suggestion....  I implemented your 2nd suggestion but I want to see if there is a more reliable / faster way to do it. I will try the JS solution and see 🙂

Helpful resources

Announcements
Power Automate News & Announcements

Power Automate News & Announcements

Keep up to date with current events and community announcements in the Power Automate community.

Community Calls Conversations

Community Calls Conversations

A great place where you can stay up to date with community calls and interact with the speakers.

Power Automate Community Blog

Power Automate Community Blog

Check out the latest Community Blog from the community!

Top Solution Authors
Top Kudoed Authors
Users online (3,732)