no unique UI selector, is there a way to read the screen to select element?

[Anonymous]

hi, I am trying to automate a process in Agile PLM to download attachments. I can select the text field and do a search, but when it return, there are multiple fields in the table :

I want to click on the dot with the row that status is active (2nd row in the picture), is there a way to do that? I tried to extract data from web but I dont see a table to loop....

And lets say if I can do this, the next step is to select a pdf file to download. When I add the UI elements the selectors for the jpg link and pdf link are identical, is there a way to read the text and click the link that contains pdf?

[MichaelAnnis]

There is image based processing.  The warning I provide is that for image based processing to work, anyone using the flow should be using the same sized monitor and the same display ratio as when the bot was originally set, and should not be remoted in.  The slightest variances can throw off image identification.  With that said, it is very accurate, very fast, and works very well when it seems like there is no other option. 

[Anonymous]

I have been testing with the OCR method but it is pretty slow (takes a couple sec to find and move mouse to the text) and it doesnt always find it on the first try too

[MichaelAnnis]

One of the biggest helps to OCR is to create the Tesseract Engine first, use a multiplier of 3 and 3.  This basically zooms in on all the letters increasing the accuracy greatly; however, it makes it slower, because now there are more pixels to go through.  I have seen other flows that are intuitive in this, but they take longer to build, so they will start with a multiplier of 1 and 1, if they don't find what they need, they have an error subflow that increases the multiplier to 2 and 2, and tries again.  

[Daniel_Pa]

You could just use Move mouse to Image, and set an offset to the left, so you hit the link. Simple and pretty fast. You might have to tweak the tolerance.

Be aware that the size of the window must be static then.

[vamsi_varanasi]

@Anonymous - here you have the below options

1. if this is a web application, go to HTML code and trace the path

2.use the send keys options ( same like key board and their short cuts )*

3. use the mouse click options along with X and Y co-ordinates**

4.use mouse and send keys option

5. OCR with mouse or key board options

* - works perfect in unattended mode and risk in attended mode

** - as mentioned above by @Daniel_Pa  , should be screen size static always. 

[tkuehara · Accepted solution]

Hi @Anonymous 

Short answer: yes, you can.

For those kind of solutions, I'd go for JS. Idk how familiar you are with JS but here it goes:

The above is a sample code (changed the element identifiers and comments). You add it in this action:

Then you apply the same logic for the second part of your code: if you get "true" from this JavaScript then it found the desired row and clicked the "dot" icon; in other words, you can now proceed to identify the pdf row and click the respective download button (assuming you have only one PDF per table).

Long answer: you could technically do that with elements as well, but it is a hassle: basically, you'll need to map at least two elements (one for the label/text and another one for the links... for the first part, and then another pair for the second table) also you'll need to extract the number of rows either by retrieving them as a data table to get the number of rows or... yes, using JS to do the same. Finally, you edit the elements to make them dynamic, so when you iterate the rows it updates the element's reference. Example:

[Anonymous]

Hi @tkuehara 

Thanks for the JavaScript suggestion....  I implemented your 2nd suggestion but I want to see if there is a more reliable / faster way to do it. I will try the JS solution and see 🙂