Image Recognition Issue

(1) Share

Report

Posted on by alee001

407

When using the "Move Mouse to Image" function to select multiple Excel worksheets, I found that if I input the two states of the (Code )worksheet in the settings (unselected or selected), it will distinguish between the two images and execute them. However, if the worksheet is in the cursor position (mouse arrow) or waiting state, it cannot recognize it. I also have no way to capture this state and execute it. (See image)

Categories:

Power Automate Mobile App

I have the same question (0)

All responses (4)

Answers (0)

Sort by

Suggested answer

Valantis 6,735 on at

Like
a
(1)

Report
Copy link

Link copied!
Hi @alee001,

The root cause is that Move mouse to image only matches what's visibly on screen and the moment your mouse pointer sits over the tab, it changes the tab's pixels enough that the saved image no longer matches. That's why your selected/unselected images work fine, but anything involving the hover state is unreliable.

For Excel specifically, there's no need to click the tab visually at all. Use Set active Excel worksheet instead it activates a sheet by name or index and is far more reliable than image matching.

If you do need to interact with the tab visually, capture it as a UI element and use Click UI element in window with a text-based selector (e.g. the tab name "Code"). That's built on UIA/MSAA selectors and doesn't break when the cursor changes the appearance.

Launch/Attach Excel → Set active Excel worksheet = "Code" the cleanest approach

Capture the tab as a UI element → Click UI element in window with a text-based selector

Image matching only viable for stable, cursor-free states. You can add a capture delay and tune the tolerance, but it'll never be dependable when the mouse is sitting on top of the element

Best regards,

Valantis

✅ If this helped solve your issue, please Accept as Solution so others can find it quickly.

❤️ If it didn’t fully solve it but was still useful, please click “Yes” on “Was this reply helpful?” or leave a Like :).

🏷️ For follow-ups @Valantis.

📝 https://valantisond365.com/

Was this reply helpful? Yes No
alee001 407 on at

Like
a
(1)

Report
Copy link

Link copied!

Valantis, thank you for your response.

I certainly used the "move mouse to image" operation only when absolutely necessary, and I know how to solve this problem, such as by moving the mouse pointer.

However, avoiding the problem isn't the root solution. I don't know if this is a bug, but perhaps there should be a mechanism to handle this situation. I hope raising this issue will bring it to everyone's attention.

Was this reply helpful? Yes No
Suggested answer

Valantis 6,735 on at

Like
a
(1)

Report
Copy link

Link copied!

Hi @alee001 ,

I agree, moving the mouse away is only a workaround.

Microsoft’s docs say image-based actions match visible pixels, and they note a known issue where clicks can land in the wrong place; the documented mitigation is to lower the tolerance on “Move mouse to image.”

I couldn’t find any setting that makes it ignore hover/cursor state when the cursor changes the control’s appearance. For a supported production path, the guidance is to favor Excel actions or UI elements and treat image-based matching as a fallback.

I’d suggest submitting/voting on an idea and opening a support ticket with clear repro steps and screenshots. Sources: Known issue + tolerance guidance

Best regards,

Valantis

✅ If this helped solve your issue, please Accept as Solution so others can find it quickly.

❤️ If it didn’t fully solve it but was still useful, please click “Yes” on “Was this reply helpful?” or leave a Like :).

🏷️ For follow-ups @Valantis.

📝 https://valantisond365.com/

Was this reply helpful? Yes No
alee001 407 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi Valantis

Strangely, when executing N different but identical files in a loop, using "move mouse to image" to execute a button in a VBA userform, even if the mouse cursor is at the same button location, it still repeatedly executes the button action.

While PAD has multiple methods to execute the same process, each with its advantages and disadvantages, it's incomprehensible that the same situation has different execution possibilities.

Was this reply helpful? Yes No