Answered

Unable to detect UI elements in Power App

(0) Share

Report

Posted on by DavidValencia

Hi there,

I've noticed that it isn't possible to detect any UI element from a web app (Power App), therefore making it impossible to automate a Power App that is already created. Am I doing something wrong or there is really no way to interact automatically with Power Apps?

I've tried with mouse and keyboard input but there are calendars and comboboxes.

Categories:

Power Automate Desktop

I have the same question (0)

All responses (11)

Answers (1)

Sort by

DavidValencia 23 on at

Like
a
(0)

Report
Copy link

Link copied!

I must mention that im trying to capture text boxes and what not using the Desktop version of Power Automate. Just for clearing any doubts.

Was this reply helpful? Yes No
Verified answer

LogicSmith 225 on at

Like
a
(0)

Report
Copy link

Link copied!

Do you have the browser extensions installed and enabled? Have you tried multiple browsers? Sometimes you will find pages or sites where various tools absolutely cannot see inside. When that happens, about all you can do is try to drive it by keystrokes and absolute mouse actions. OCR can also help in this situation. You can also ask the app's creator what they can do.

Was this reply helpful? Yes No
DavidValencia 23 on at

Like
a
(0)

Report
Copy link

Link copied!

Thanks for replying LogicSmith,

I have the browser extension enabled only in Chrome and have not tried different browsers, which i'm going to do right now with their respective extensions. I tried the keystrokes and mouse inputs but haven't even thought about using the OCR features, which is a pretty well thought idea i must say.

On the other hand i've been in touch with the creator of the Power App but he really doesn't know anything about the problem or how to solve it.

I'll update the case's state as soon as i can.

Was this reply helpful? Yes No
VJR 7,635 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @DavidValencia

Not all elements or applications on the screen are going to be detectable/automatable.

What is your complete use case that you are detecting elements of PowerApps from Power Automate Desktop?

Was this reply helpful? Yes No
DavidValencia 23 on at

Like
a
(0)

Report
Copy link

Link copied!

Hello there @VJR ,

I don't fully understand your question but i'll answer what i think you're refering to. I basically want to automate a form located within a Power App developed by my company, and I can't locate anything. I was thinking about what other user replied to me about using the ocr features for detecting certain buttons in some comboboxes and to change the calendars presents in the power app for text entries.

Was this reply helpful? Yes No
VJR 7,635 on at

Like
a
(0)

Report
Copy link

Link copied!

My question was why are you interacting with a PowerApps app via the Power Automate Desktop..What are you exactly trying to do? There could be a better design.

Yes of course your options are adding UI elements (not working for you), Sendkeys, OCR, but these are not always going to be reliable however you can try it out and if it works then good for you.

Was this reply helpful? Yes No
DavidValencia 23 on at

Like
a
(0)

Report
Copy link

Link copied!

I was trying to interact with our Power App because it was such a good oportunity to automate something. The app is an Inventory and our company is constanly registering devices. All the infrastructure was already created when i arrived but i thought that automating the process would save us so much time on the long run.

Was this reply helpful? Yes No
LogicSmith 225 on at

Like
a
(0)

Report
Copy link

Link copied!

Sadly, I have no idea what is behind a Power App... I know of their existence, but I have no experience there. In the early days of Windows and test automation tools, we rapidly found we could not "see inside" of anything made by certain development tools (Borland Pascal, Silverlight). It certainly may be the same thing exists with other newer tools such as Python. Same thing exists with some web sites.

Sometimes it is possible to find bizarre workarounds, so don't give up if it's important to you. I see that some of the actions involve running PowerShell or VBScripts, which can probably do things that PAD cannot. Keep trying, but you might see if there's a Power Apps forum here (I'm sure I've seen it somewhere) and ask them.

Was this reply helpful? Yes No
LogicSmith 225 on at

Like
a
(0)

Report
Copy link

Link copied!
If you do go the route of looking for images on the screen, here's a couple of tips...

Force the screen into a known resolution when you start your app or browser. Maximize the app. We used 1920x1080 as that seems to be pretty standard.
Operating systems matter. A VM hosted by Windows Server handles fonts on the screen just a bit differently from Windows 7/10/11, even though they may look the same to most users.
If it's in a browser, those matter too. Edge and Chrome use the same render engine, while Firefox and Internet Explorer use different ones.
You may have to find a way to manage your images if you use a lot of them.

Was this reply helpful? Yes No
DavidValencia 23 on at

Like
a
(0)

Report
Copy link

Link copied!

It indeed is important to me as it's my first contact with power apps and power automate. Screen tips are already done, i'll use a computer just for automating this task, probably change it to a virtual one. I only use Chrome and I'll have to work my way around some comboboxes without being able to access de UI elements which will be a headache for sure.

Was this reply helpful? Yes No