Move mouse to text on Screen ( OCR ), worked yesterday now dont work at all?

(1) Share

Report

Posted on by DanielOlsson

111

Hello

I'm stuck with this failure, any ideas are welcome.

Yesterday this action worked perfectly, today, it does not work at all. Tested on the same text that it worked on yesterday, on two different installations that are exact. Nothing has changed, I copied the flow and tested; to exclude any of the hidden issues that sometimes can break a flow.

Error says it cannot find the text. =(

I have further tried to improve the action by setting X1 Y1 and X2 Y2 cords to help but to no avail.

Ideas welcome

Regards

Daniel Olsson

Categories:

Power Automate Desktop

I have the same question (0)

All responses (8)

Answers (1)

Sort by

businesstaken 26 on at

Like
a
(1)

Report
Copy link

Link copied!

ocr recogniziont working erratically, pretty much an hit or miss feature.
I've experienced same thing and gave up using it, I can't understand why same word in the same page doesn't get recognized anymore

Was this reply helpful? Yes No
DanielOlsson 111 on at

Like
a
(0)

Report
Copy link

Link copied!

Have you found an alternative? I will log a support ticket. if the static text can be read and later not, the product has a flaw. OCR is very important.

Was this reply helpful? Yes No
tkuehara 667 on at

Like
a
(2)

Report
Copy link

Link copied!

Hi @DanielOlsson

I've worked with OCR in the past and noticed that, sometimes, changing the multiplier helps you achieve a better result:
With this in mind, I created a "retry" structure: if multiplier 1 doesn't bring any result, I'd then retry OCR with multiplier 2. Then, if failed again, I'd try multiplier 3. Also, when you reduce the screen area to search for a text, the failure rate is lower than trying to read the entire screen (and a lot faster as well). The disadvantage is that if you have a text appearing in different parts of the screen each time you run a flow then you'll need a very complex solution. At that time, I had to create another "retry structure level" with different sections of the screen: for example, if the search failed for the entire screen for all three multipliers, then I divided the screen by two (first vertically and then horizontally) and tried to look for the same text again in the first half and then the second half for all three multipliers again, if it has failed again then I'd further divide the screen in three or four parts (depends on the resolution) and look for the text again. This made the solution a lot more reliable (say it, from a failure rate of 1 in 10 to 1 in 40), sacrificing speed and increasing complexity though.
Also notice that if you are working with Citrix, the images presented on the screen might look the same but they can be sufficiently different for the OCR engine to not recognize it. This is because of an "adaptive compression" mechanism Citrix uses that sends a different resolution depending on the currently available GPU power, network bandwith and other parameters. Long story short: when using Citrix, expect it will fail from time to time because there will always be a "new image" it never sent before.

Was this reply helpful? Yes No
DanielOlsson 111 on at

Like
a
(0)

Report
Copy link

Link copied!

Thank you very much. I will set up and test this and let you know if it makes a difference. ( I bet it will ) in addition, I will share the outcome of the support ticket.

Was this reply helpful? Yes No
Verified answer

DanielOlsson 111 on at

Like
a
(3)

Report
Copy link

Link copied!

The solution :

I had a call with Microsoft support engineer Sunil Barate and he explained a few useful things. Below is an example of a flow and how you can set it up if it does not seem to work.

1. Add config on the start ORC engine action to 2 2. This enables zooming in to help read the text.

2. The Get Window action seems to help the ORC to work better, even if we can see the window without the Get action in my flow it didn't work. It's a pretty good idea to focus the windows that the OCR should read, so I will always set that in the future.

3. A wait after starting ORC engine and focusing window, to help it all be in the correct state before moving the mouse by OCR start is as well recommended, the 5 sec is excessive but as it works you could test 4 3 2 etc.

4. To know the ORC can read the text, you can add an extract text with ORC and check that the text you look for is in the output variable.

Good luck!

Was this reply helpful? Yes No
tkuehara 667 on at

Like
a
(1)

Report
Copy link

Link copied!

Thanks for sharing!

Was this reply helpful? Yes No
phuft 14 on at

Like
a
(3)

Report
Copy link

Link copied!

I have had this same issue many times. I use the recognize text step to look for a specific word on the page and then click it. I have done all the steps recommended below and the step still randomly fails--magnification is at 2/2, I have the flow focusing on the desired window, I have a wait before the OCR step, but it still randomly breaks.

The only thing that temporarily fixes the issue is to paste a copy of the failing OCR step and delete the original step. I have no idea why this works, but so far it has worked every time. I can only assume it is some type of caching issue related to engineers trying to speed up inefficient code.

This will give me another random number of days of the OCR working, after which I will have to repeat to get it working again. I have sat through the entire flow multiple times and watched the text I am seeking show up clearly on the screen but not get recognized.

I have had the exact same issue with clicking on UI Elements--they worked for a while but then stopped and the only way to repair them was to select the UI element again in the failing step--even though the UI element had not changed.

I really hope Microsoft is able to fix this product-breaking issue, because without consistently working dynamic UI element/text recognition Power Automate is almost useless.

Was this reply helpful? Yes No
Bretzeli 13 on at

Like
a
(1)

Report
Copy link

Link copied!

Hello,

Nice others notice....

I am a Senior System Engineer IT and have been working for over 25 years with MS products, We are MS partner and reseller.

The OCR function of flow/Power Automate is absolute unuseable. If the OCR should work better in a zoom 2x2 why don't they intergrate in their code direct?

This all looks nice at the beginning but is just the worst terrible MS solution/Code we have ever seen in 20 years.

We do NOT recommend to automate anything business related in Enterprise with this solution.
You just move cost from regular backoffice to IT 2nd level if this keeps on working like this.

Worst is that it's run fully from their cloud. My PA on W11 PRO before W10 22H2 fully patches crashed again and there is no solution to recover the changes we made. Thought it may get better with W11.

Was this reply helpful? Yes No