Problem statement:
Some applications do not allow capturing UI elements or adding selectors.
In such cases, Image based recording comes to the rescue.
For example, you would like to capture the title of the below message box and the actual error message that is at a certain position from an anchor ie; the exclamation sign in the below box.
Or you want to extract a text next to another text but UI elements and selectors are not possible.
Solution: Text based and Image based anchors.
This can be easily done with Image based recording in Power Automate Desktop.
Steps to follow:
1. Click the recorder button on the top.
Be noted that in the newer versions of PAD, there is one single recorder button for Web, Desktop and Image based recording.
2. Toggle to enable the Image based recording
3. Click the Record button shown on the above screen.
Right click on the text you want to extract and click "Extract text from image".
4. Power Automate Desktop will ask you to perform 2 steps
In Step 1: you select the text area that you want to extract using the rounded mouse pointer as shown below
In Step 2: The outcome of Step 1 will be a red border around the text area to be extracted.
Now, use the circular mouse pointer to select the anchor area which is the Exclamation mark image.
Once both the steps are correctly done, PAD will highlight both the regions and show it under the preview image button.
The blue colored region is the anchor area and the red colored is the text to be extracted.
If the regions look right then proceed further.
5. On clicking the Done button, this gives an "Extract text with OCR" action in PAD showing the preview of the anchor area and the co-ordinates of the text to be extracted from the anchor.
Tip: The anchor can also be a text instead of an image.
For instance, in the below example, ID: is a Text anchor and you want to extract the value next to it ie; 1025.
Use it wherever UI elements and selectors are not available.
6. On running the Flow, the output text will be generated in the variable called Extracted Text.
Side notes:
What we did above was for extracting the text next to the exclamation mark.
This exact same procedure can be followed to extract the wordings shown in the title ie; "Error Message".
This is very handy in case you need to have your flow differentiate between Warnings and Error Messages.
Once you get the Extracted Text variable as "Error Message" you can use Text functions available in PAD and check its value.
Tip: Use an anchor area (image or text), that is not expected to change in future.
