cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
DanielOlsson
Helper II
Helper II

Move mouse to text on Screen ( OCR ), worked yesterday now dont work at all?

Hello

 

I'm stuck with this failure, any ideas are welcome. 

Screenshot_4.png

Yesterday this action worked perfectly, today, it does not work at all.  Tested on the same text that it worked on yesterday, on two different installations that are exact. Nothing has changed, I copied the flow and tested; to exclude any of the hidden issues that sometimes can break a flow. 

 

Error says it cannot find the text. =(

I have further tried to improve the action by setting X1 Y1 and X2 Y2 cords to help but to no avail. 

 

Ideas welcome

 

Regards

Daniel Olsson

 

1 ACCEPTED SOLUTION

Accepted Solutions
DanielOlsson
Helper II
Helper II

The solution :  

 

I had a call with Microsoft support engineer Sunil Barate and he explained a few useful things.  Below is an example of a flow and how you can set it up if it does not seem to work.

 

1. Add config on the start ORC engine action to 2  2.  This enables zooming in to help read the text.

 

2. The Get Window action seems to help the ORC to work better, even if we can see the window without the Get action in my flow it didn't work. It's a pretty good idea to focus the windows that the OCR should read, so I will always set that in the future.

 

3. A wait after starting ORC engine and focusing window, to help it all be in the correct state before moving the mouse by OCR start is as well recommended, the 5 sec is excessive but as it works you could test 4 3 2 etc.

 

4. To know the ORC can read the text, you can add an extract text with ORC and check that the text you look for is in the output variable.

 

DanielOlsson_0-1627311065185.png

 

Good luck!

View solution in original post

6 REPLIES 6
businesstaken
Frequent Visitor

ocr recogniziont working erratically, pretty much an hit or miss feature.

I've experienced same thing and gave up using it, I can't understand why same word in the same page doesn't get recognized anymore

Have you found an alternative? I will log a support ticket.  if the static text can be read and later not, the product has a flaw.  OCR is very important.

Hi @DanielOlsson 

 

I've worked with OCR in the past and noticed that, sometimes, changing the multiplier helps you achieve a better result:

tkuehara_2-1626989701976.png

With this in mind, I created a "retry" structure: if multiplier 1 doesn't bring any result, I'd then retry OCR with multiplier 2. Then, if failed again, I'd try multiplier 3. Also, when you reduce the screen area to search for a text, the failure rate is lower than trying to read the entire screen (and a lot faster as well). The disadvantage is that if you have a text appearing in different parts of the screen each time you run a flow then you'll need a very complex solution. At that time, I had to create another "retry structure level" with different sections of the screen: for example, if the search failed for the entire screen for all three multipliers, then I divided the screen by two (first vertically and then horizontally) and tried to look for the same text again in the first half and then the second half for all three multipliers again, if it has failed again then I'd further divide the screen in three or four parts (depends on the resolution) and look for the text again. This made the solution a lot more reliable (say it, from a failure rate of 1 in 10 to 1 in 40), sacrificing speed and increasing complexity though.

Also notice that if you are working with Citrix, the images presented on the screen might look the same but they can be sufficiently different for the OCR engine to not recognize it. This is because of an "adaptive compression" mechanism Citrix uses that sends a different resolution depending on the currently available GPU power, network bandwith and other parameters. Long story short: when using Citrix, expect it will fail from time to time because there will always be a "new image" it never sent before.

DanielOlsson
Helper II
Helper II

Thank you very much. I will set up and test this and let you know if it makes a difference.  ( I bet it will ) in addition, I will share the outcome of the support ticket. 

DanielOlsson
Helper II
Helper II

The solution :  

 

I had a call with Microsoft support engineer Sunil Barate and he explained a few useful things.  Below is an example of a flow and how you can set it up if it does not seem to work.

 

1. Add config on the start ORC engine action to 2  2.  This enables zooming in to help read the text.

 

2. The Get Window action seems to help the ORC to work better, even if we can see the window without the Get action in my flow it didn't work. It's a pretty good idea to focus the windows that the OCR should read, so I will always set that in the future.

 

3. A wait after starting ORC engine and focusing window, to help it all be in the correct state before moving the mouse by OCR start is as well recommended, the 5 sec is excessive but as it works you could test 4 3 2 etc.

 

4. To know the ORC can read the text, you can add an extract text with ORC and check that the text you look for is in the output variable.

 

DanielOlsson_0-1627311065185.png

 

Good luck!

Thanks for sharing!

Helpful resources

Announcements
Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

New Ideas Forum MPA.jpg

A new place to submit your Ideas for Power Automate

Announcing a new way to share your feedback with the Power Automate Team.

Carousel_PP_768x460_Wave2 (1).png

2022 Release Wave 2 Plan

Power Platform release plan for the 2022 release wave 2 describes all new features releasing from October 2022 through March 2023.

MPA Virtual Workshop Carousel 768x460.png

Register for a Free Workshop

Learn to digitize and optimize business processes and connect all your applications to share data in real time.

365 EduCon 768x460.png

Microsoft 365 EduCon

Join us for two optional days of workshops and a 3-day conference, you can choose from over 130 sessions in multiple tracks and 25 workshops.

Top Solution Authors
Top Kudoed Authors
Users online (3,205)